-- Logs begin at Wed 2022-10-12 01:51:01 UTC, end at Wed 2022-10-12 02:54:16 UTC. -- Oct 12 02:15:48.395704 dv-cinder-849143-14-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 12 02:15:48.404877 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Oct 12 02:15:48 2022] *** Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: nodename: dv-cinder-849143-14-iscsi Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: machine: x86_64 Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: clock source: unix Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: pcre jit disabled Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: detected number of CPU cores: 4 Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: current working directory: / Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: detected binary path: /usr/bin/uwsgi-core Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: your processes number limit is 47680 Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: your memory page size is 4096 bytes Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: detected max file descriptor number: 2048 Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: lock engine: pthread robust mutexes Oct 12 02:15:48.407174 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: thunder lock: enabled Oct 12 02:15:48.409537 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 12 02:15:48.409537 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 12 02:15:48.446244 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: Python main interpreter initialized at 0x56403ad189c0 Oct 12 02:15:48.446244 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: python threads support enabled Oct 12 02:15:48.446244 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: your server socket listen backlog is limited to 100 connections Oct 12 02:15:48.446244 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: your mercy for graceful operations on workers is 90 seconds Oct 12 02:15:48.446583 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: mapped 403077 bytes (393 KB) for 2 cores Oct 12 02:15:48.446663 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: *** Operational MODE: preforking *** Oct 12 02:15:48.446932 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: *** uWSGI is running in multiple interpreter mode *** Oct 12 02:15:48.446932 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: spawned uWSGI master process (pid: 81350) Oct 12 02:15:48.447123 dv-cinder-849143-14-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 12 02:15:48.449226 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: spawned uWSGI worker 1 (pid: 81364, cores: 1) Oct 12 02:15:48.449226 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: spawned uWSGI worker 2 (pid: 81365, cores: 1) Oct 12 02:15:48.449226 dv-cinder-849143-14-iscsi devstack@keystone.service[81350]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 12 02:15:52.686016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 12 02:15:52.686016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: from cryptography.utils import int_from_bytes Oct 12 02:15:52.686810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 12 02:15:52.686810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: from cryptography.utils import int_from_bytes Oct 12 02:15:52.687711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 12 02:15:52.687711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: from cryptography.utils import int_from_bytes Oct 12 02:15:52.687818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 12 02:15:52.687818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: from cryptography.utils import int_from_bytes Oct 12 02:15:52.736075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 12 02:15:52.736253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 12 02:15:52.736436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] command line args: [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 12 02:15:52.736529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 12 02:15:52.736635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 12 02:15:52.736747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 12 02:15:52.736837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] command line args: [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 12 02:15:52.736911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ================================================================================ {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 12 02:15:52.737109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 12 02:15:52.737196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] admin_token = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ================================================================================ {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 12 02:15:52.737352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] config_dir = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] admin_token = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] config_source = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] config_dir = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] debug = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.737955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] config_source = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] debug = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.738928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_config_append = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.739958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_config_append = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_dir = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_file = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_dir = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_file = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.740926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.741975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.742933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.23/identity {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] publish_errors = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.743996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.23/identity {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] publish_errors = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.744888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.745032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.745160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.745279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] standard_threads = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.745672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.745803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.745921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] standard_threads = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] use_journal = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] use_json = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.746978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] use_journal = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] use_stderr = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] use_json = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] use_syslog = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] use_stderr = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] use_syslog = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.747832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.747954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 12 02:15:52.748079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.748203 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.748329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.748730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.748857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.748974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.749089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.749206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.749343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.749463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.750918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.751874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.752995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.753893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.754985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755347 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.755947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.756935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.757997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.758889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.759967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.760980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.761961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.762930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.763983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.764840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.765082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.765191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.765342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.765342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.766917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.767989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.768969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.769941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.770912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.771935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.772914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.773994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.774838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.external = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.external = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.775819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.password = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] auth.token = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.password = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] auth.token = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.776943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.777908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.778994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.779881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.780909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.781885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.782978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.783939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.784919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.785952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786187 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.786906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.787880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.788899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.789998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.790945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.791999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.792992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.793928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.794771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.795939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.796932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.797922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.798970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.799940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.800945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.801989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.802957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.803927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804407 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.804940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.805907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.806998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.807959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.808899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] role.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] role.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] role.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.809991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] role.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.810951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.811905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.812024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.812625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.812766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.812886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.813958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.814907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.815995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.816877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.817974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.818932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.819887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.820992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.821969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.connection = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.822951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.connection = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.823992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.824897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.825986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81364) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 12 02:15:52.826927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 12 02:15:52.826927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81365) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 12 02:15:52.850211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.850438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.850660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.851196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.851428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.851654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.858837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.860386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.887754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.887982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.888527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.888788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.940169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.940503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.988556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.988805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.989165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:52.989402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.143935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.144066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.144149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.144279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.144364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.144516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.166761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.167988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.168115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.168199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.168338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.179579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.179961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.192555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.192808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.192947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193698 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.193908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81364) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.194037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.194248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81365) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 12 02:15:53.210367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.210575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 12 02:15:53.210660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.210859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 12 02:15:53.210942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.211102 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 12 02:15:53.211185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.211370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.211453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 12 02:15:53.211627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.211721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.211881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.211967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.212136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.212217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.212398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.221732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.222839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.229506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.229837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.232390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.232673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.232785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.232977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.233885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.234592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.234679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.234862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.234945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.235929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.259363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.260852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.261137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.262826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.264294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.264577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.265894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.266182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.266464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.270625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.270912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.271193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.272868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.273158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.277555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.277842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.278835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.279135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.279424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.279711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.279984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.280257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.280527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.280822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.284046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.284340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.284629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.284942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.285222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.285498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.285773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.286042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.296910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.297196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.301078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.301461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.301741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.304308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.304605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.305548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.308733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.308873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.309036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.312606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.313721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.315852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.316152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.316440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.316742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.316828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.317033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.317320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.319954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.320253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.320537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.320847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.321126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.321402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.333674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.333965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.334233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.334493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.341226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.341381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.341517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.341786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.342047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.342918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.344471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.344781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.347597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.347884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.348175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.348460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.348871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.348958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.350419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.351935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.352222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.354958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.355243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.355524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.355805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.356200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.360057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.360344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.360627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.360930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.361311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.361600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.361885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.367349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.367630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.367907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.368184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.368559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.368861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.369140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.378102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.381029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.383925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.385291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.385377 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.386837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.387120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.387530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.388183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.391239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.392672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.393971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.394152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.394433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.394843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.395427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.398333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.398731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.399017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.399295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.401202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.402670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.405519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.406441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.406725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.407002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.407681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.407981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.408262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.414890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.415171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81365) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.415256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.415448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.415538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.415727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.416006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.416280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.416561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.416867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.417151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.417420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.417694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.417965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.418242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.418508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.418784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81365) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.422876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81364) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 12 02:15:53.423158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.423441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.423725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.423997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.424266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.424539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.424838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.425107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.425381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.425656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.425925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.426228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.426505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81364) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 12 02:15:53.453325 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56403ad189c0 pid: 81365 (default app) Oct 12 02:15:53.456522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af0626-34d2-43ed-abe8-66a4ffa057fc None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:15:53.456834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af0626-34d2-43ed-abe8-66a4ffa057fc None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:15:53.457110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af0626-34d2-43ed-abe8-66a4ffa057fc None None] PATH_INFO: `/v3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:15:53.460163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56403ad189c0 pid: 81364 (default app) Oct 12 02:15:53.461239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1/1] 10.222.0.23 () {50 vars in 791 bytes} [Wed Oct 12 02:15:53 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:00.154802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:00.155201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:00.155347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:00.160140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1/2] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:00 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:00.165224 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:00.165666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:00.165828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:00.236148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: INFO dbcounter [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] Registered counter for database keystone Oct 12 02:16:00.254946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG oslo_db.sqlalchemy.engines [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81365) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 12 02:16:00.261108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writer thread running {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 12 02:16:00.436446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG passlib.handlers.bcrypt [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81365) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 12 02:16:00.436763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG passlib.handlers.bcrypt [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81365) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 12 02:16:00.494326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:00.495907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:00.620042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 2/3] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:00 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 456 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 12 02:16:01.045761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:01.116145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: INFO dbcounter [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] Registered counter for database keystone Oct 12 02:16:01.137000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG oslo_db.sqlalchemy.engines [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81364) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 12 02:16:01.143583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writer thread running {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 12 02:16:01.261525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957474b2-5f64-48f9-acf3-e252e529493f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:01.266319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:01.266931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:01.267210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:01.267481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:01.275598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:01.328282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING keystone.common.rbac_enforcer.enforcer [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 12 02:16:01.328636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073b5b88-86b2-473b-aadf-dfea824eb3d0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:01.338143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 2/4] 10.222.0.23 () {58 vars in 1324 bytes} [Wed Oct 12 02:16:01 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:16:01.344663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:01.361396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e71581a-6de3-498d-adfd-10bf2e0cf11c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:01.365862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:01.366961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:01.367242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:01.367515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] PATH_INFO: `/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:01.369146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] RBAC: Authorizing `identity:get_project(project_id=1b152cd4fded41fa8ba95182a5dc98e4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:01.422960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING keystone.common.rbac_enforcer.enforcer [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 12 02:16:01.423313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9514603-5396-44e5-9e4a-2fa2b93a5e67 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:01.427876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 3/5] 10.222.0.23 () {56 vars in 1207 bytes} [Wed Oct 12 02:16:01 2022] GET /identity/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:03.506700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a30137-4efa-4641-b5a7-1c193af0178d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:03.507198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a30137-4efa-4641-b5a7-1c193af0178d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:03.507298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a30137-4efa-4641-b5a7-1c193af0178d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:03.511392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 3/6] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:03.516918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:03.517324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:03.517465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:03.537217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:03.538747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:03.547089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 4/7] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:03 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 12 02:16:03.813243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a30137-4efa-4641-b5a7-1c193af0178d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:03.825061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a30137-4efa-4641-b5a7-1c193af0178d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:03.829224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dffa07-b47b-4c72-9cee-77fc8715154c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:03.829822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dffa07-b47b-4c72-9cee-77fc8715154c admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:03.830098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dffa07-b47b-4c72-9cee-77fc8715154c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:03.830368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dffa07-b47b-4c72-9cee-77fc8715154c admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:03.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dffa07-b47b-4c72-9cee-77fc8715154c admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:03.839626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dffa07-b47b-4c72-9cee-77fc8715154c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:03.848678 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 4/8] 10.222.0.23 () {58 vars in 1324 bytes} [Wed Oct 12 02:16:03 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:16:03.853747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:03.863564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fc9ab7-f12d-4cdb-a2ea-5d615ff28c03 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:03.867918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0171ef-888f-441d-9816-f280fa00cd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:03.868538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0171ef-888f-441d-9816-f280fa00cd1a admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:03.868849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0171ef-888f-441d-9816-f280fa00cd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:03.869125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0171ef-888f-441d-9816-f280fa00cd1a admin admin] PATH_INFO: `/v3/users/1bca8e4c760d498693a8dcfc0e178249` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:03.870440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0171ef-888f-441d-9816-f280fa00cd1a admin admin] RBAC: Authorizing `identity:get_user(user_id=1bca8e4c760d498693a8dcfc0e178249)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:03.872291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0171ef-888f-441d-9816-f280fa00cd1a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:03.874119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 5/9] 10.222.0.23 () {56 vars in 1198 bytes} [Wed Oct 12 02:16:03 2022] GET /identity/v3/users/1bca8e4c760d498693a8dcfc0e178249 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:06.034744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b106db7-aa53-4009-aa72-5abed20105b2 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.035312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b106db7-aa53-4009-aa72-5abed20105b2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.035312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b106db7-aa53-4009-aa72-5abed20105b2 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.036596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 5/10] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:06.041640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.042071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.042297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.059976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:06.061166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:06.097391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40ab65a-be7d-4fb5-98ec-655c045cee24 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.097689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40ab65a-be7d-4fb5-98ec-655c045cee24 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.097943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40ab65a-be7d-4fb5-98ec-655c045cee24 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.099498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 6/11] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:06.104601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.104897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.105166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.143560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 6/12] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:06.147776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG passlib.handlers.bcrypt [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81364) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 12 02:16:06.148171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG passlib.handlers.bcrypt [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81364) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 12 02:16:06.196747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:06.198302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:06.295802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 7/13] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:06.412553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.436555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:06.519518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2656b-211b-4778-99d1-66ae62c03453 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:06.552050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:06.552610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.552911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.553183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.560269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.565318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:06.574226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:06.575246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:06.576076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 7/14] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:16:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:06.580944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.659688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5498760-b0e1-41b5-8120-5a11c530b50c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:06.661945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:06.662194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:06.681537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53859bd-9e89-442d-a7f0-3ceb22b52a08 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:06.698795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:06.699663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.699940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.700211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.701221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:06.702650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:06.713360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.sql.core [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 12 02:16:06.713360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 12 02:16:06.713360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [parameters: {'id': '413003de01ce4f5ba3d272c62ecc1683', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 12 02:16:06.713360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81364) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 12 02:16:06.715251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:06.715850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.716124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.716388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.717072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:06.717368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:06.718771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:06.730397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 8/15] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:16:06.731736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 12 02:16:06.732670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application yield resource Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.rollback() Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.force_reraise() Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise self.value Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application session.commit() Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 12 02:16:06.733612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._prepare_impl() Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.flush() Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._flush(objects) Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application compat.raise_( Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application flush_context.execute() Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rec.execute(self) Oct 12 02:16:06.734958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 12 02:16:06.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:16:06.738191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [parameters: {'id': '413003de01ce4f5ba3d272c62ecc1683', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 12 02:16:06.739599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:06.740682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.741859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 8/16] 10.222.0.23 () {60 vars in 1137 bytes} [Wed Oct 12 02:16:06 2022] POST /identity/v3/domains => generated 190 bytes in 173 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:06.742653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.742739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.835623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e95af0f-e3e8-48ff-9843-8d9ad08e147c None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:06.838828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc276a-1bce-46fe-a390-8e34b4d812bc None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:06.868641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:06.869338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.869623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.869892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] PATH_INFO: `/v3/users/1bca8e4c760d498693a8dcfc0e178249` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.871028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] RBAC: Authorizing `identity:get_user(user_id=1bca8e4c760d498693a8dcfc0e178249)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:06.871825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:06.872481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:06.872671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:06.872981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:06.873255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:06.874832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 9/17] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:06 2022] GET /identity/v3/users/1bca8e4c760d498693a8dcfc0e178249 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:06.880207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.895134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:06.895787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:06.897563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:16:06.913900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:06.914959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:16:06.915877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:16:06.915877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:16:06.915877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:06.915877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 9/18] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:06.920745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:06.942632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:06.981951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b552e1-51cb-46b0-b289-26673f2aaac5 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:07.017023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:07.017635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:07.017911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:07.018182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:07.019272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:07.020753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:07.022432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 10/19] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:07.024453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc00bca-3b42-400c-9905-ac4604b5eb18 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:07.027900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:07.059892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:07.060502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:07.060812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:07.061086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:07.061726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:07.062562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:07.063955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7862091-b5e2-437d-8ad1-62446a1263b4 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:07.081496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 10/20] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:16:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:07.111089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:07.133610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721d2e1-a733-4461-ae36-b90ea021583a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:07.169637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8513c2-8256-4d6f-bf4a-208ebfeefc98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:07.170316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8513c2-8256-4d6f-bf4a-208ebfeefc98 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:07.170593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8513c2-8256-4d6f-bf4a-208ebfeefc98 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:07.170863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8513c2-8256-4d6f-bf4a-208ebfeefc98 None admin] PATH_INFO: `/v3/domains/default/users/1bca8e4c760d498693a8dcfc0e178249/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:07.173494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8513c2-8256-4d6f-bf4a-208ebfeefc98 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=1bca8e4c760d498693a8dcfc0e178249, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:07.174964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8513c2-8256-4d6f-bf4a-208ebfeefc98 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:07.291048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 11/21] 10.222.0.23 () {58 vars in 1361 bytes} [Wed Oct 12 02:16:07 2022] PUT /identity/v3/domains/default/users/1bca8e4c760d498693a8dcfc0e178249/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:16:09.695731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d045d2a-740c-4852-8318-1e5bd33d9b82 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:09.696307 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d045d2a-740c-4852-8318-1e5bd33d9b82 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:09.696592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d045d2a-740c-4852-8318-1e5bd33d9b82 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:09.698129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 11/22] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:09.715195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca8b1d6-38ab-482c-afef-309f2902f57a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:09.715705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca8b1d6-38ab-482c-afef-309f2902f57a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:09.715809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca8b1d6-38ab-482c-afef-309f2902f57a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:09.745271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-8ca8b1d6-38ab-482c-afef-309f2902f57a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:09.746828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8ca8b1d6-38ab-482c-afef-309f2902f57a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:09.830593 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a23da-6f82-40eb-82a5-cebe24f8eb2b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:09.830912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a23da-6f82-40eb-82a5-cebe24f8eb2b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:09.831143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a23da-6f82-40eb-82a5-cebe24f8eb2b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:09.832722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 12/23] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:09.838001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:09.838285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:09.838563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:09.843821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 12/24] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:09.877740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:09.878963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:09.906833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958dcdf-34fe-4172-bc47-7842dd362dc1 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:09.907136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958dcdf-34fe-4172-bc47-7842dd362dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:09.907393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958dcdf-34fe-4172-bc47-7842dd362dc1 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:09.911954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 13/25] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:09.927955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:09.928226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:09.928501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:09.969787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:09.970922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:09.995138 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 13/26] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:10.074606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:10.084785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 14/27] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:10.372065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:10.384709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:10.388846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:10.509908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb174c8-d474-4193-a595-08526d2a9c6b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:10.519528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd26b20-b62d-4b31-8df0-e1df1b20bd05 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:10.557018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:10.557658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:10.557945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:10.558218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:10.561675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:10.562816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:10.563095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:10.563369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:10.564049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:10.565459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:10.572391 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:10.574576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-81478595-be75-48c6-88f1-170cb293aa41 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:16:10.593656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:10.594577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:16:10.595387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:16:10.595387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:16:10.595387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:10.595387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 15/28] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:10.596333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e6f8f-5f9d-48f9-83b4-ff8c0bc8ce9f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:10.596684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e6f8f-5f9d-48f9-83b4-ff8c0bc8ce9f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:10.597052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e6f8f-5f9d-48f9-83b4-ff8c0bc8ce9f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:10.598350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 16/29] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:10.599118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56e6f8f-5f9d-48f9-83b4-ff8c0bc8ce9f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:10.660859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:10.697671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 14/30] 10.222.0.23 () {60 vars in 1131 bytes} [Wed Oct 12 02:16:10 2022] POST /identity/v3/roles => generated 225 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:10.699274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:10.725197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56e6f8f-5f9d-48f9-83b4-ff8c0bc8ce9f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:10.762449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e5313-1240-4b99-b178-50cb7d0f53b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:10.763301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e5313-1240-4b99-b178-50cb7d0f53b6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:10.763702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e5313-1240-4b99-b178-50cb7d0f53b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:10.764043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e5313-1240-4b99-b178-50cb7d0f53b6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:10.778082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e5313-1240-4b99-b178-50cb7d0f53b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-6e6e5313-1240-4b99-b178-50cb7d0f53b6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:10.784488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:10.785374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:10.786128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 17/31] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:16:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:10.787476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:10.787829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:10.788173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:10.808028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:10.809389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:10.827418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:10.861715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab22ec0-4f43-484e-a8b2-97f65348e945 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:10.898634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:10.899677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:10.900186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:10.900657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:10.901549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:10.902052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:10.903816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:10.917689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 15/32] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:16:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:10.918984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:10.952188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 18/33] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:10.960829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:11.028364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5baca-5360-46e9-8225-50ea37ca4dfa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.061231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:11.079020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.080014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.080501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.081005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.081887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:11.082412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.084017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.086350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:11.110273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 16/34] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:11.116642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:11.199835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5e0eb1-2368-4634-97ef-592abc6d8e05 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.264038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2bfa2e-b0d3-4780-83bf-290d38620563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.264685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2bfa2e-b0d3-4780-83bf-290d38620563 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.265033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2bfa2e-b0d3-4780-83bf-290d38620563 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.265310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2bfa2e-b0d3-4780-83bf-290d38620563 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.265993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2bfa2e-b0d3-4780-83bf-290d38620563 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.267533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2bfa2e-b0d3-4780-83bf-290d38620563 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.315204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b09266-9009-4112-8499-7c0540c55c98 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.321145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 19/35] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:16:10 2022] POST /identity/v3/projects => generated 301 bytes in 369 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:11.322581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9764c83e-be2b-49ee-a2e4-87779dd42c3b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.322925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9764c83e-be2b-49ee-a2e4-87779dd42c3b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.323206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9764c83e-be2b-49ee-a2e4-87779dd42c3b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.326439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 20/36] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:11.330934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ce9840-7b3a-41ab-a928-bf1c2c7fb3a6 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.331221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ce9840-7b3a-41ab-a928-bf1c2c7fb3a6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.331507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ce9840-7b3a-41ab-a928-bf1c2c7fb3a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.380142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-47ce9840-7b3a-41ab-a928-bf1c2c7fb3a6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:11.382452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-47ce9840-7b3a-41ab-a928-bf1c2c7fb3a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:11.388099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.389260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.389710 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.390184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] PATH_INFO: `/v3/users/1bca8e4c760d498693a8dcfc0e178249` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.391930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] RBAC: Authorizing `identity:get_user(user_id=1bca8e4c760d498693a8dcfc0e178249)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.393784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.396060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 17/37] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:11 2022] GET /identity/v3/users/1bca8e4c760d498693a8dcfc0e178249 => generated 247 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:11.397183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:11.472808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:11.472972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 21/38] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:11.472972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2967e-1a51-472e-901c-784ca945f251 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.472972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2967e-1a51-472e-901c-784ca945f251 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.472972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2967e-1a51-472e-901c-784ca945f251 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.472972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 22/39] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:11.472972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2967e-1a51-472e-901c-784ca945f251 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:11.505663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:11.534314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb37e129-b165-4efe-a390-f02af10c8b15 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.574273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b08831-db86-44bd-9677-0312415f845d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.574891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b08831-db86-44bd-9677-0312415f845d None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.575168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b08831-db86-44bd-9677-0312415f845d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.575439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b08831-db86-44bd-9677-0312415f845d None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.576107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b08831-db86-44bd-9677-0312415f845d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.577589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b08831-db86-44bd-9677-0312415f845d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.599275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 18/40] 10.222.0.23 () {60 vars in 1131 bytes} [Wed Oct 12 02:16:11 2022] POST /identity/v3/roles => generated 219 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:11.600445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.600803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.603336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.610414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2967e-1a51-472e-901c-784ca945f251 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.623391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:11.625521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:11.716655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 19/41] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:11.759052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.759646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.759896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.760171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.760946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:11.761342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.762868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.764735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 23/42] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:11.777746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:11.824278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:11.887680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601f1b9-d693-4bc6-bf0f-e3eb010830cd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.892919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:11.908058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dda3174-65dd-4893-a604-8e8d5ff2c577 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:11.933511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.934350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.934539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.935824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.935824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.937436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.950113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:11.950732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:11.951013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:11.951286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:11.952020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.domain.id=default, user.id=1bca8e4c760d498693a8dcfc0e178249) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:11.952327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:11.954572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:11.956842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 20/43] 10.222.0.23 () {60 vars in 1131 bytes} [Wed Oct 12 02:16:11 2022] POST /identity/v3/roles => generated 223 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:16:11.962078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 24/44] 10.222.0.23 () {56 vars in 1322 bytes} [Wed Oct 12 02:16:11 2022] GET /identity/v3/role_assignments?user.id=1bca8e4c760d498693a8dcfc0e178249&scope.domain.id=default&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 530 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:11.989566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:12.110840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc59ca4-f4d2-4c63-807a-c9daf4efc311 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:12.128289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:12.159251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f168266a-b26d-4295-80a9-3d788b1e2972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:12.159889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168266a-b26d-4295-80a9-3d788b1e2972 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:12.160163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168266a-b26d-4295-80a9-3d788b1e2972 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:12.160432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168266a-b26d-4295-80a9-3d788b1e2972 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:12.161603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168266a-b26d-4295-80a9-3d788b1e2972 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:12.163043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168266a-b26d-4295-80a9-3d788b1e2972 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:12.164601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 21/45] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:12.170504 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:12.277134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f3251-25d6-4053-9fa4-47608e0f2f8e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:12.310347 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495bf97b-de58-419f-bd54-e4d1d9074394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:12.310954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495bf97b-de58-419f-bd54-e4d1d9074394 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:12.311241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495bf97b-de58-419f-bd54-e4d1d9074394 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:12.311518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495bf97b-de58-419f-bd54-e4d1d9074394 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:12.312369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495bf97b-de58-419f-bd54-e4d1d9074394 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:12.313868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495bf97b-de58-419f-bd54-e4d1d9074394 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:12.339311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 25/46] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:16:12 2022] POST /identity/v3/projects => generated 312 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:14.458724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ef32b-1220-44b2-bbf3-d59505718e66 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:14.459202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ef32b-1220-44b2-bbf3-d59505718e66 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:14.459287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ef32b-1220-44b2-bbf3-d59505718e66 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:14.460607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 22/47] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:14.465727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:14.466136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:14.466302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:14.484965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:14.486147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:14.488056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:14.552612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 26/48] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:14.822280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ef32b-1220-44b2-bbf3-d59505718e66 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:14.919932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ef32b-1220-44b2-bbf3-d59505718e66 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:14.951011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcfc5ab-1b40-4dbf-928c-8e3897cb9d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:14.951621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfc5ab-1b40-4dbf-928c-8e3897cb9d04 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:14.951897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfc5ab-1b40-4dbf-928c-8e3897cb9d04 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:14.952165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfc5ab-1b40-4dbf-928c-8e3897cb9d04 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:14.953802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfc5ab-1b40-4dbf-928c-8e3897cb9d04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:14.955222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfc5ab-1b40-4dbf-928c-8e3897cb9d04 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:14.956765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 23/49] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:14.962339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:15.058621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ac667b-9f57-4e0b-ba3a-74b1ce5be532 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:15.092493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a7e854-ea49-48c5-8f77-d09f64941d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:15.093068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a7e854-ea49-48c5-8f77-d09f64941d3e None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:15.093349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a7e854-ea49-48c5-8f77-d09f64941d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:15.093623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a7e854-ea49-48c5-8f77-d09f64941d3e None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:15.094285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a7e854-ea49-48c5-8f77-d09f64941d3e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:15.095763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a7e854-ea49-48c5-8f77-d09f64941d3e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:15.105136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:15.121402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 27/50] 10.222.0.23 () {60 vars in 1140 bytes} [Wed Oct 12 02:16:14 2022] POST /identity/v3/projects => generated 298 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:17.247971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38deec-0c8f-4a21-9299-300a85ba3c8a None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:17.248475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38deec-0c8f-4a21-9299-300a85ba3c8a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:17.248585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38deec-0c8f-4a21-9299-300a85ba3c8a None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:17.250774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 24/51] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:17.255864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:17.256266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:17.256422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:17.274349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:17.275535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:17.340443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 28/52] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:17.609467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38deec-0c8f-4a21-9299-300a85ba3c8a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:17.656967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:17.709157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38deec-0c8f-4a21-9299-300a85ba3c8a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:17.741343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef626f7-3a97-4648-bbcf-6842e143fa77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:17.741969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef626f7-3a97-4648-bbcf-6842e143fa77 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:17.742251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef626f7-3a97-4648-bbcf-6842e143fa77 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:17.742521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef626f7-3a97-4648-bbcf-6842e143fa77 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:17.743625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef626f7-3a97-4648-bbcf-6842e143fa77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:17.745096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef626f7-3a97-4648-bbcf-6842e143fa77 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:17.746648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 25/53] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:17.752447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:17.851204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f43e9f-b4e7-401a-baa2-be8bed685c87 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:17.870570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:17.885375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be0c9d3-e690-4acb-9d0b-d149763fb403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:17.885977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be0c9d3-e690-4acb-9d0b-d149763fb403 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:17.886258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be0c9d3-e690-4acb-9d0b-d149763fb403 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:17.886532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be0c9d3-e690-4acb-9d0b-d149763fb403 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:17.887174 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be0c9d3-e690-4acb-9d0b-d149763fb403 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:17.888658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be0c9d3-e690-4acb-9d0b-d149763fb403 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:17.915642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 29/54] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:17 2022] POST /identity/v3/users => generated 275 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:21.130738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfb4d84-74ea-470b-b802-937e5e18594b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.131662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfb4d84-74ea-470b-b802-937e5e18594b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.132130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfb4d84-74ea-470b-b802-937e5e18594b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.133695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 26/55] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:21.139067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8948a0b-5ecf-4d53-afce-7c7f1227ad4c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.140385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8948a0b-5ecf-4d53-afce-7c7f1227ad4c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.140913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8948a0b-5ecf-4d53-afce-7c7f1227ad4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.164922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f8948a0b-5ecf-4d53-afce-7c7f1227ad4c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:21.166111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f8948a0b-5ecf-4d53-afce-7c7f1227ad4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:21.188033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcceec1-ba13-46ae-8592-e86b0d0f7714 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.188607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcceec1-ba13-46ae-8592-e86b0d0f7714 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.189150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcceec1-ba13-46ae-8592-e86b0d0f7714 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.191163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 27/56] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:21.196870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a681c4-4cd6-4125-abae-bd3a459ac62c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.197384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a681c4-4cd6-4125-abae-bd3a459ac62c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.197897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a681c4-4cd6-4125-abae-bd3a459ac62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.217643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-07a681c4-4cd6-4125-abae-bd3a459ac62c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:21.219423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-07a681c4-4cd6-4125-abae-bd3a459ac62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:21.291133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 30/57] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:21.297660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 28/58] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:21.425662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f412d9-0b22-4817-8c1c-16c65184d64b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.425866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f412d9-0b22-4817-8c1c-16c65184d64b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.426336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f412d9-0b22-4817-8c1c-16c65184d64b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.427886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 31/59] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:21.433352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.433710 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.434061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.457403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:21.459231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:21.508042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:21.536241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 29/60] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:21.620089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f412d9-0b22-4817-8c1c-16c65184d64b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:21.702787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:21.759547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f412d9-0b22-4817-8c1c-16c65184d64b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:21.780920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:21.813292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc734dd-2a46-47d1-b633-bda3dfddb92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:21.813931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc734dd-2a46-47d1-b633-bda3dfddb92c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.814208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc734dd-2a46-47d1-b633-bda3dfddb92c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.814481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc734dd-2a46-47d1-b633-bda3dfddb92c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.828552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc734dd-2a46-47d1-b633-bda3dfddb92c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-3dc734dd-2a46-47d1-b633-bda3dfddb92c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:21.835746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:21.836579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:21.837858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:21.838923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 32/61] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:16:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:21.840286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a2bc6-bae1-4493-bfc1-7bc9ef378881 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.840599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a2bc6-bae1-4493-bfc1-7bc9ef378881 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.845536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a2bc6-bae1-4493-bfc1-7bc9ef378881 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.848783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 33/62] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:21 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:21.848783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6a2bc6-bae1-4493-bfc1-7bc9ef378881 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:21.853377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf348f21-b87b-4d53-a9bf-be8467374f58 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:21.890220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c4b451-fb8c-45e2-9206-2bb0baae6a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:21.890861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c4b451-fb8c-45e2-9206-2bb0baae6a60 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:21.891144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c4b451-fb8c-45e2-9206-2bb0baae6a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:21.891413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c4b451-fb8c-45e2-9206-2bb0baae6a60 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:21.904044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c4b451-fb8c-45e2-9206-2bb0baae6a60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:21.964073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6a2bc6-bae1-4493-bfc1-7bc9ef378881 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.003155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.003813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.004090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.004359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.017232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:22.022421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:22.023375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:22.024134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 34/63] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:16:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:22.025859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-41c4b451-fb8c-45e2-9206-2bb0baae6a60 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:22.026583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:22.027430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:22.028863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 30/64] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:22.029502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.029502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.029621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.051431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:22.052684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:22.062657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.082655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.134284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 31/65] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:22.135177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.148576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd454979-9ec2-4c69-8993-ba111e4f66a2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.186198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.187045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.187488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.187907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.188572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:22.188893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.190302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.199194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 35/66] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.199783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.253935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580cb643-bdc8-4843-a3b4-bfe8b5acfd75 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.291503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.292114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.292398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.292670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.293364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:22.293667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.295045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cc0944-96b6-4432-a4f0-ac76a3a4ae7e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.297848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.304154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 32/67] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.306660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d425434-54e8-4e73-885d-cbbc974fb1e9 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.307126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d425434-54e8-4e73-885d-cbbc974fb1e9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.307832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d425434-54e8-4e73-885d-cbbc974fb1e9 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.309718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 33/68] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:22.310613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d425434-54e8-4e73-885d-cbbc974fb1e9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.312009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6f0e3b-f49f-4582-b3ff-3677fff5aba4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.341394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.348485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.349111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.349415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.349662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.350316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] RBAC: Adding query filter params (name=member) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:22.350621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.352597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.360855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 36/69] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.361737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.428630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d425434-54e8-4e73-885d-cbbc974fb1e9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.462378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62663ec-718f-42f2-b639-32f12ea53b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.463009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62663ec-718f-42f2-b639-32f12ea53b06 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.463169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb480f3-5f4e-487c-a176-122ac8e8347f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.463286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62663ec-718f-42f2-b639-32f12ea53b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.463553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62663ec-718f-42f2-b639-32f12ea53b06 None admin] PATH_INFO: `/v3/users/1bca8e4c760d498693a8dcfc0e178249` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.464677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62663ec-718f-42f2-b639-32f12ea53b06 None admin] RBAC: Authorizing `identity:get_user(user_id=1bca8e4c760d498693a8dcfc0e178249)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.466141 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62663ec-718f-42f2-b639-32f12ea53b06 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.467715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 34/70] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/users/1bca8e4c760d498693a8dcfc0e178249 => generated 247 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.468857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.469140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.469427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.486617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:22.487816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:22.498295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.499665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.499939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.500214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] PATH_INFO: `/v3/users/e66d8b7ddd4d4982adf0584a255beaf1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.513114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] RBAC: Authorizing `identity:get_user(user_id=e66d8b7ddd4d4982adf0584a255beaf1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.514527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.516126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 37/71] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/users/e66d8b7ddd4d4982adf0584a255beaf1 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.516902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.528876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.558769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 35/72] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:22.559612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.585994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.622537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b68aa-da28-4e3e-8a5a-97e434625833 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.657697 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.658330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.658659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.658922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] PATH_INFO: `/v3/users/e66d8b7ddd4d4982adf0584a255beaf1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.659971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] RBAC: Authorizing `identity:get_user(user_id=e66d8b7ddd4d4982adf0584a255beaf1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.661454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.667404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 38/73] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/users/e66d8b7ddd4d4982adf0584a255beaf1 => generated 275 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.667549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0786ed8-b8fa-4e19-8d5b-a55007a2f6e6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.667641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.703109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.703769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.704013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.704283 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.716429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:22.722691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:22.723537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:22.724303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 36/74] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:22.725805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.760826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.786647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7f8bd-a442-4179-8ed4-695bcdfb06cc None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.812951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:22.822471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.823187 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.823472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.823743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.825054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.826364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.828517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 39/75] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.829309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.835687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e6e98a-9b52-4f27-9253-4c9608d4a1aa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.872214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.873013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.873296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.875531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.877019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.878677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 37/76] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.879432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.931738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a511e7-f776-4ff6-9b0f-0e1340ce7c89 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.965019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:22.965651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:22.965928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:22.966200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:22.967304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:22.969329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:22.975792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 40/77] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:22.975792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:22.982810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c33075a-88f9-4aae-abae-dfb9410cb840 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:22.986439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:23.017202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.017816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.018101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.018378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.019035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:23.019341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.020717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.028908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 38/78] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:23.029781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.046839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:23.075327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0722be1-744e-4054-ac32-26ea229c71cb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.109643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.110278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.110559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.110835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.111954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.113466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.115321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 41/79] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:23.115963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.136125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc41c754-52dc-4da4-9fce-f8c2d90580c7 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.169988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.170650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.170935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.171212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/1bca8e4c760d498693a8dcfc0e178249/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.173175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] RBAC: Authorizing `identity:create_grant(project_id=239dfc9cbb164dc89983358a6d8450bd, user_id=1bca8e4c760d498693a8dcfc0e178249, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.174614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.189053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 39/80] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:23 2022] PUT /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/1bca8e4c760d498693a8dcfc0e178249/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:23.189830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.206441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:23.229068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cce59c-3728-4ab2-b303-cec4577d85a6 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.268376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.269297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.269683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.270064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/e66d8b7ddd4d4982adf0584a255beaf1/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.272456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] RBAC: Authorizing `identity:create_grant(project_id=239dfc9cbb164dc89983358a6d8450bd, user_id=e66d8b7ddd4d4982adf0584a255beaf1, role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.274103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.278360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:23.289764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 42/81] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:23 2022] PUT /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/e66d8b7ddd4d4982adf0584a255beaf1/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:23.297804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.314486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3549707b-75a6-499c-9a1a-bf3dd0b85817 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.358372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.359081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.359462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.359789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/e66d8b7ddd4d4982adf0584a255beaf1/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.362834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] RBAC: Authorizing `identity:create_grant(project_id=239dfc9cbb164dc89983358a6d8450bd, user_id=e66d8b7ddd4d4982adf0584a255beaf1, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.364175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.381132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 40/82] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:23 2022] PUT /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/e66d8b7ddd4d4982adf0584a255beaf1/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:23.381976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.442303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d745a-72e7-47ce-80c6-4c1c8e5bbec1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.497942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.498613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.499727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.500003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] PATH_INFO: `/v3/users/e66d8b7ddd4d4982adf0584a255beaf1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.501319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] RBAC: Authorizing `identity:get_user(user_id=e66d8b7ddd4d4982adf0584a255beaf1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.503544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.505836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 43/83] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:23 2022] GET /identity/v3/users/e66d8b7ddd4d4982adf0584a255beaf1 => generated 275 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:23.512642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.520051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d843b2-bdcf-4e3e-9507-0063f0e2ba9f None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.525145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:23.573059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.574474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.575013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.575460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.576310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.578771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.601566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:23.614416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 41/84] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:23 2022] POST /identity/v3/users => generated 289 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:23.653685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229c80e-2b49-47ef-8a48-7dfa935e8754 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.695862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c8a46-1395-4ff1-ba7c-8644a360909d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.696964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c8a46-1395-4ff1-ba7c-8644a360909d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.697386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c8a46-1395-4ff1-ba7c-8644a360909d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.697657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c8a46-1395-4ff1-ba7c-8644a360909d None admin] PATH_INFO: `/v3/projects/1455f317818a4158ae6331403b2bfe1d` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.699113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c8a46-1395-4ff1-ba7c-8644a360909d None admin] RBAC: Authorizing `identity:get_project(project_id=1455f317818a4158ae6331403b2bfe1d)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.700679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c8a46-1395-4ff1-ba7c-8644a360909d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.702711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 44/85] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:23 2022] GET /identity/v3/projects/1455f317818a4158ae6331403b2bfe1d => generated 312 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:23.708575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:23.835221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a267012-6148-476d-994c-f164ac8cdad2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:23.880242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27b1c80-4cbc-42f4-b18f-333e7ab7e3d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:23.881263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b1c80-4cbc-42f4-b18f-333e7ab7e3d4 None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:23.881733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b1c80-4cbc-42f4-b18f-333e7ab7e3d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:23.882173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b1c80-4cbc-42f4-b18f-333e7ab7e3d4 None admin] PATH_INFO: `/v3/projects/1455f317818a4158ae6331403b2bfe1d/users/e66d8b7ddd4d4982adf0584a255beaf1/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:23.884282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27b1c80-4cbc-42f4-b18f-333e7ab7e3d4 None admin] RBAC: Authorizing `identity:create_grant(project_id=1455f317818a4158ae6331403b2bfe1d, user_id=e66d8b7ddd4d4982adf0584a255beaf1, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:23.885981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27b1c80-4cbc-42f4-b18f-333e7ab7e3d4 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:23.900386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 42/86] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:23 2022] PUT /identity/v3/projects/1455f317818a4158ae6331403b2bfe1d/users/e66d8b7ddd4d4982adf0584a255beaf1/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:26.109086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80ddd1-4a76-415e-af95-e141b409138f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.110112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80ddd1-4a76-415e-af95-e141b409138f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.110607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80ddd1-4a76-415e-af95-e141b409138f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.112179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 45/87] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:26.121882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7695d2e-839b-48f1-9cdd-50fa46c37330 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.122798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7695d2e-839b-48f1-9cdd-50fa46c37330 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.123325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7695d2e-839b-48f1-9cdd-50fa46c37330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.140483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9de1b4-a4c0-47eb-a3d2-e70b71629db4 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.143770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9de1b4-a4c0-47eb-a3d2-e70b71629db4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.143770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9de1b4-a4c0-47eb-a3d2-e70b71629db4 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.143770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 46/88] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:26.148218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5829866-fc72-49d1-9a32-9ed0d7f08a73 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.148502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5829866-fc72-49d1-9a32-9ed0d7f08a73 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.148807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5829866-fc72-49d1-9a32-9ed0d7f08a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.151590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d7695d2e-839b-48f1-9cdd-50fa46c37330 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:26.153023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d7695d2e-839b-48f1-9cdd-50fa46c37330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:26.168712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a5829866-fc72-49d1-9a32-9ed0d7f08a73 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:26.170192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a5829866-fc72-49d1-9a32-9ed0d7f08a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:26.221084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:26.259540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 43/89] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:26.290566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 47/90] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:26.558535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48a2f32-004c-4e23-aa7d-78562ce8252b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.559215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48a2f32-004c-4e23-aa7d-78562ce8252b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.559788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48a2f32-004c-4e23-aa7d-78562ce8252b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.561394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 44/91] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:26.569200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.569502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.569756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.582803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:26.594317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:26.596160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:26.676032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48a2f32-004c-4e23-aa7d-78562ce8252b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:26.678011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 48/92] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:26.791795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:26.838427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48a2f32-004c-4e23-aa7d-78562ce8252b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:26.893708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:26.895063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:26.895535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:26.895996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:26.910974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:26.916229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:26.917085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:26.917856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:26.924482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 45/93] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:16:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:26.929703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:26.992158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dff5217-7696-46e1-a7aa-b817d3a9d753 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.042873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.077953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d0ee7b-61f7-4182-ae00-bcfaf4c870fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.078609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d0ee7b-61f7-4182-ae00-bcfaf4c870fd None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.078888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d0ee7b-61f7-4182-ae00-bcfaf4c870fd None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.079163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d0ee7b-61f7-4182-ae00-bcfaf4c870fd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.094227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.110583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d0ee7b-61f7-4182-ae00-bcfaf4c870fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c3d0ee7b-61f7-4182-ae00-bcfaf4c870fd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:27.115524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:27.116372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:27.117601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:27.120865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 49/94] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:27.122575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e781b-0c66-4f9c-be6a-6c04b0203946 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.122868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e781b-0c66-4f9c-be6a-6c04b0203946 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.123151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e781b-0c66-4f9c-be6a-6c04b0203946 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.124765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 50/95] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:27.125786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb03a1-30fd-40fe-8af5-8edbaa2718ac None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.126061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb03a1-30fd-40fe-8af5-8edbaa2718ac None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.126337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb03a1-30fd-40fe-8af5-8edbaa2718ac None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.127520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 51/96] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:27 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:27.128963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbb03a1-30fd-40fe-8af5-8edbaa2718ac None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.132450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56057a1c-4eef-416a-ad8a-5fae7c96f66b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.175789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.176396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.176679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.176978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.177635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:27.178052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:27.179583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd16b137-f9b1-4140-ac4e-5fc6e9220254 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:27.189336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 46/97] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:27.190483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d93da1-7c54-4293-a831-ae880c1cc323 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.190767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d93da1-7c54-4293-a831-ae880c1cc323 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.191040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d93da1-7c54-4293-a831-ae880c1cc323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.209551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-10d93da1-7c54-4293-a831-ae880c1cc323 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:27.210747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-10d93da1-7c54-4293-a831-ae880c1cc323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:27.241503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbb03a1-30fd-40fe-8af5-8edbaa2718ac None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.277809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.278426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.278707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.278984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.279645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:27.279951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:27.288524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 47/98] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:27.288668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:27.288783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.288783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.288783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.297626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 52/99] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:27.298283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.307491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:27.308674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:27.339565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.369034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.393620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 48/100] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:27.402852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.441758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016b814-129e-486d-b534-31ebfe0bc5c4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.511370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.512018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.512296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.512567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.537530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:27.543046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:27.543989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:27.544803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 53/101] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:27.550697 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.562646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2397e85c-39fc-4079-b1c5-9f464c34609f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.601355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.602318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.602790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.603419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] PATH_INFO: `/v3/users/1bca8e4c760d498693a8dcfc0e178249` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.605252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] RBAC: Authorizing `identity:get_user(user_id=1bca8e4c760d498693a8dcfc0e178249)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:27.606959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:27.608800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 49/102] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/users/1bca8e4c760d498693a8dcfc0e178249 => generated 247 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:27.609702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.648232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.697179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edf5cff-780a-4df4-afd2-1864759be279 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.714658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.721691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031093f-b512-4fa0-a4e0-a90669081bfd None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.732142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.732890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.733258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.734076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] PATH_INFO: `/v3/users/e66d8b7ddd4d4982adf0584a255beaf1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.735202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] RBAC: Authorizing `identity:get_user(user_id=e66d8b7ddd4d4982adf0584a255beaf1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:27.736674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:27.738418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 54/103] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/users/e66d8b7ddd4d4982adf0584a255beaf1 => generated 275 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:27.739006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.759455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.760067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.760342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.760618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.761320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:27.761624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:27.762983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:27.771243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 50/104] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:27.771243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.839701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69221d02-9da1-470d-871a-7a1952c2b5eb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.862994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.872203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bb3ca6-5493-43f3-91bf-b523ad02efd9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:27.872784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988783cb-6bef-4074-9cbb-da3159c8ff34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.873426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988783cb-6bef-4074-9cbb-da3159c8ff34 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.873971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988783cb-6bef-4074-9cbb-da3159c8ff34 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.874793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988783cb-6bef-4074-9cbb-da3159c8ff34 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.884666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988783cb-6bef-4074-9cbb-da3159c8ff34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-988783cb-6bef-4074-9cbb-da3159c8ff34 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:27.891272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:27.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:27.892933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 55/105] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:27.894144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a834a5-bb8a-447a-a77a-d655bc99c9ac None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.894144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a834a5-bb8a-447a-a77a-d655bc99c9ac None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.894144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a834a5-bb8a-447a-a77a-d655bc99c9ac None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.895412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 56/106] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:27 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:27.895923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a834a5-bb8a-447a-a77a-d655bc99c9ac None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:27.910814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:27.911361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:27.911642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:27.911913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:27.944276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:27.979233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:27.980731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:27.998981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a834a5-bb8a-447a-a77a-d655bc99c9ac None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.031528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.032144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.032418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.032685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.042370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:28.048137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:28.049057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 57/107] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:28.050086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.068396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.078749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 51/108] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:28.079579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.154289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f2c17-2434-4d1b-8931-a32e579d5f40 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.161695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.179298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e166-0186-47c4-bf7d-eb0a99b8019c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.189334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d6ea49-a1ae-4832-b365-f54714c3a7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.189942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d6ea49-a1ae-4832-b365-f54714c3a7f9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.190218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d6ea49-a1ae-4832-b365-f54714c3a7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.190486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d6ea49-a1ae-4832-b365-f54714c3a7f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.212466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.213114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.213397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.213667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] PATH_INFO: `/v3/users/e66d8b7ddd4d4982adf0584a255beaf1` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.214790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] RBAC: Authorizing `identity:get_user(user_id=e66d8b7ddd4d4982adf0584a255beaf1)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.216243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.218389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 52/109] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/users/e66d8b7ddd4d4982adf0584a255beaf1 => generated 275 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:28.219156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.256037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d6ea49-a1ae-4832-b365-f54714c3a7f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.257552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d6ea49-a1ae-4832-b365-f54714c3a7f9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.281977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.323933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d300746f-39c1-4a63-9b4b-a4d372aa612c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.356888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.357493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.357770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.358042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.358966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 58/110] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:28.359249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:28.359552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.360213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.360495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.360785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.360976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.370873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 53/111] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:28.371673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.377619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.383899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:28.385115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:28.449693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 59/112] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:28.450348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.470297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3459dc06-4554-482d-9959-72b964a77737 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.491718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.505179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.505790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.506065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.506336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.506987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:28.507287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.508648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.517314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 54/113] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:28.518105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.554444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e4c1fc-e4eb-4ba1-b2ba-b15a302ffd4d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.588489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.589353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.589770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.590169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.591408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.593083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.594993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 60/114] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:28.595913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.602424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.633730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9e3e0-b8d1-4e93-ad2e-4aa53839ed42 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.672381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.673345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.674309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.674782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.701337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4eb0ae-bec2-4703-98f9-212f4c3e9f31 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.740665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.748508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.749609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.750017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.750294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.752009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.753760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.760572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 61/115] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:28.760572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.760770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.760770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.836718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.870488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 55/116] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:28.870827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d623b80-6380-4c80-88aa-77ccf0ba4dc2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:28.871343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.910246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:28.910884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:28.911170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:28.911440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] PATH_INFO: `/v3/users/b5d01144d8624ca1a699142879d9f980` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:28.925590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] RBAC: Authorizing `identity:get_user(user_id=b5d01144d8624ca1a699142879d9f980)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:28.927074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:28.928917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 62/117] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/users/b5d01144d8624ca1a699142879d9f980 => generated 289 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:28.929594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:28.974748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:28.994051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c76af-b926-4523-a842-3c2ff5a3a036 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.032271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.032957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.033239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.033508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] PATH_INFO: `/v3/users/e66d8b7ddd4d4982adf0584a255beaf1` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.034625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] RBAC: Authorizing `identity:get_user(user_id=e66d8b7ddd4d4982adf0584a255beaf1)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.036075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.036464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ec9278-30fc-4103-8373-db78c38c3550 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.037682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 56/118] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/users/e66d8b7ddd4d4982adf0584a255beaf1 => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.038437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.070138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.075817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.076427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.076722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.077070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.078309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=239dfc9cbb164dc89983358a6d8450bd, user.id=1bca8e4c760d498693a8dcfc0e178249) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:29.078618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.080063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.085755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 63/119] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:28 2022] GET /identity/v3/role_assignments?user.id=1bca8e4c760d498693a8dcfc0e178249&scope.project.id=239dfc9cbb164dc89983358a6d8450bd&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.086524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.146117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b9aa65-3a2c-4d08-8bbe-42c7df3ffedf None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.187800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.188554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.188835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.189013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.190876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.192755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.194757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 57/120] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.195874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.218477 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b58bc4-526d-47aa-bc1a-89687f4df607 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.223536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.257840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.258358 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.258638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.258937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.259636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] RBAC: Adding query filter params (role.id=7914a50fd85645bd870c4f7502de5b02, scope.project.id=239dfc9cbb164dc89983358a6d8450bd, user.id=e66d8b7ddd4d4982adf0584a255beaf1) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:29.259939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.261441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.268288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 64/121] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/role_assignments?user.id=e66d8b7ddd4d4982adf0584a255beaf1&scope.project.id=239dfc9cbb164dc89983358a6d8450bd&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.271825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.336439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9407a8f9-301a-47fc-bf1c-356a960ce330 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.360571 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.378514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.379373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.379752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.380098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.381392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.382945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.384807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 58/122] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.385688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.417819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26544759-bb44-4b20-a5d0-f5c7e0b9ef20 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.458894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187b209-4222-4263-875b-620914254338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.459535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187b209-4222-4263-875b-620914254338 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.459819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187b209-4222-4263-875b-620914254338 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.460099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187b209-4222-4263-875b-620914254338 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.461323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8187b209-4222-4263-875b-620914254338 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.462984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8187b209-4222-4263-875b-620914254338 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.464663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 65/123] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.465439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187b209-4222-4263-875b-620914254338 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.489596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.506114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab195a-10d5-4148-b0b8-3c2a411c621d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.541251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.541778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.542058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.542325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.580341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187b209-4222-4263-875b-620914254338 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.604556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.609676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.611131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.613670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.614284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.614568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.614845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.615498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.616984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.643436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 66/124] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:16:29 2022] POST /identity/v3/projects => generated 302 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:29.644248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.715179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.730041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 59/125] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:29.736367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.766723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55b7224-a0b7-4849-82a0-b03cb618847d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.803838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.804370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.804659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.804957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.805699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] RBAC: Adding query filter params (role.id=cd7b8279c49f4441b98ba176e7e27199, scope.project.id=239dfc9cbb164dc89983358a6d8450bd, user.id=e66d8b7ddd4d4982adf0584a255beaf1) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:29.806001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.807474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.813819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 67/126] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/role_assignments?user.id=e66d8b7ddd4d4982adf0584a255beaf1&scope.project.id=239dfc9cbb164dc89983358a6d8450bd&role.id=cd7b8279c49f4441b98ba176e7e27199 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:29.814854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:29.852395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7003e42c-b93b-42fe-80e7-11724349d52e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:29.881533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:29.897228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:29.898413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:29.898413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:29.898413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/b5d01144d8624ca1a699142879d9f980/roles/127abf9263f6445bb39b087269e222f3` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:29.900388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] RBAC: Authorizing `identity:create_grant(project_id=239dfc9cbb164dc89983358a6d8450bd, user_id=b5d01144d8624ca1a699142879d9f980, role_id=127abf9263f6445bb39b087269e222f3)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:29.901995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:29.919950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 60/127] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:29 2022] PUT /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd/users/b5d01144d8624ca1a699142879d9f980/roles/127abf9263f6445bb39b087269e222f3 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:29.960259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68089817-eb3d-474d-a7ed-19b08487ef48 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:30.004843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb624647-be9b-4017-9f46-fe4f8faf038c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:30.005586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb624647-be9b-4017-9f46-fe4f8faf038c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:30.005867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb624647-be9b-4017-9f46-fe4f8faf038c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:30.006183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb624647-be9b-4017-9f46-fe4f8faf038c None admin] PATH_INFO: `/v3/projects/1455f317818a4158ae6331403b2bfe1d` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:30.007576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb624647-be9b-4017-9f46-fe4f8faf038c None admin] RBAC: Authorizing `identity:get_project(project_id=1455f317818a4158ae6331403b2bfe1d)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:30.009122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb624647-be9b-4017-9f46-fe4f8faf038c None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:30.010907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 68/128] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:29 2022] GET /identity/v3/projects/1455f317818a4158ae6331403b2bfe1d => generated 312 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:30.017612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:30.086690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:30.152223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35afb486-f00c-47b2-87a6-a12a5ddfbb8a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:30.186091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:30.186735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:30.187040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:30.187314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:30.188044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] RBAC: Adding query filter params (role.id=7914a50fd85645bd870c4f7502de5b02, scope.project.id=1455f317818a4158ae6331403b2bfe1d, user.id=e66d8b7ddd4d4982adf0584a255beaf1) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:30.188555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:30.189827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af79c91-2d20-4bb2-9df1-c96679c80dec None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:30.197851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 61/129] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:30 2022] GET /identity/v3/role_assignments?user.id=e66d8b7ddd4d4982adf0584a255beaf1&scope.project.id=1455f317818a4158ae6331403b2bfe1d&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:31.904175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844fd90-ade9-4227-9e56-4f89c25a426c None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:31.904737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844fd90-ade9-4227-9e56-4f89c25a426c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:31.904825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844fd90-ade9-4227-9e56-4f89c25a426c None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:31.906962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 69/130] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:31.912259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbed36a-d594-4a72-990f-5884d3f7b3d0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:31.912632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbed36a-d594-4a72-990f-5884d3f7b3d0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:31.912836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbed36a-d594-4a72-990f-5884d3f7b3d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:31.931970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4dbed36a-d594-4a72-990f-5884d3f7b3d0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:31.933196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4dbed36a-d594-4a72-990f-5884d3f7b3d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:32.001800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 62/131] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:32.082449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a49707c-beec-4ae2-9129-4f874c81d6b7 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.082724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a49707c-beec-4ae2-9129-4f874c81d6b7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.082998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a49707c-beec-4ae2-9129-4f874c81d6b7 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.084468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 70/132] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:32.089396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.089683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.089958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.109044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:32.110233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:32.146909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:32.177922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 63/133] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:32.270021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a49707c-beec-4ae2-9129-4f874c81d6b7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:32.367780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a49707c-beec-4ae2-9129-4f874c81d6b7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:32.378360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:32.401394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:32.402014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.402290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.402565 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.403646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:32.405210 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:32.406763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 71/134] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:32.412208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:32.447016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:32.510084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0ddea-e848-4ced-9902-174256a719b6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:32.542385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:32.542991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.543267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.543541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.544182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:32.545349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d597cf00-dc4d-4902-a81e-96055dedf633 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:32.546153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:32.566976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 64/135] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:32 2022] POST /identity/v3/users => generated 283 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:32.580633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:32.581318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.581594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.581864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.594339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:32.599894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:32.600847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:32.601857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 72/136] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:32.607418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:32.638416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:32.718303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c617d772-5b56-4acb-9541-c24f190c4036 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:32.754281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:32.754897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.755177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.755452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.756113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:32.756419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:32.757834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:32.766676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 65/137] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:32.772537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:32.814716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:32.879509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a560d737-1794-413f-ae0c-1d96cb98357a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:32.916059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:32.916769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:32.917060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:32.917334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] PATH_INFO: `/v3/users/b5d01144d8624ca1a699142879d9f980` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:32.918476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] RBAC: Authorizing `identity:get_user(user_id=b5d01144d8624ca1a699142879d9f980)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:32.919950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:32.921793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 73/138] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:32 2022] GET /identity/v3/users/b5d01144d8624ca1a699142879d9f980 => generated 289 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:32.927334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:33.032336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:33.045586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82511319-f7c0-466d-895e-7da904d3eea9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:33.082904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:33.083534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:33.083813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:33.084084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:33.155500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:33.157066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:33.250746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:33.264972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 66/139] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:33.267569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:33.378958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69755c54-13e9-4892-a9c4-0bc59f1645eb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:33.386373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:33.415258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f48e7-6e4c-410f-b9e0-0ca79b9bab48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:33.415892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f48e7-6e4c-410f-b9e0-0ca79b9bab48 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:33.416172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f48e7-6e4c-410f-b9e0-0ca79b9bab48 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:33.416492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f48e7-6e4c-410f-b9e0-0ca79b9bab48 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:33.417718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45f48e7-6e4c-410f-b9e0-0ca79b9bab48 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:33.419178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45f48e7-6e4c-410f-b9e0-0ca79b9bab48 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:33.420891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 74/140] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:33 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:33.426848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:33.529442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8588790a-7806-4f89-af80-1c949a7f527d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:33.567779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:33.568399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:33.568679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:33.568979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:33.569716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] RBAC: Adding query filter params (role.id=127abf9263f6445bb39b087269e222f3, scope.project.id=239dfc9cbb164dc89983358a6d8450bd, user.id=b5d01144d8624ca1a699142879d9f980) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:33.570024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:33.571474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9781c863-cc42-46ce-879b-bf4b7bc1d68f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:33.578056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 67/141] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:33 2022] GET /identity/v3/role_assignments?user.id=b5d01144d8624ca1a699142879d9f980&scope.project.id=239dfc9cbb164dc89983358a6d8450bd&role.id=127abf9263f6445bb39b087269e222f3 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:34.828807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c2f7e1-c9de-4b90-916b-8b1f101e9ade None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:34.829331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c2f7e1-c9de-4b90-916b-8b1f101e9ade None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:34.829331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c2f7e1-c9de-4b90-916b-8b1f101e9ade None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:34.830830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 75/142] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:34.835926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d18d64-a84e-47d4-a026-1752e45f80e5 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:34.836312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d18d64-a84e-47d4-a026-1752e45f80e5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:34.836491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d18d64-a84e-47d4-a026-1752e45f80e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:34.854676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-96d18d64-a84e-47d4-a026-1752e45f80e5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:34.855913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-96d18d64-a84e-47d4-a026-1752e45f80e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:34.910623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:34.928690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 68/143] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:34.978916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9dc1e8-2933-47de-965f-d73abc7ec5d7 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:34.979233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9dc1e8-2933-47de-965f-d73abc7ec5d7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:34.979460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9dc1e8-2933-47de-965f-d73abc7ec5d7 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:34.981047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 76/144] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:34.986780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f42ce-352e-46f1-8e07-a914e61e3d0c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:34.987078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f42ce-352e-46f1-8e07-a914e61e3d0c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:34.987355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f42ce-352e-46f1-8e07-a914e61e3d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.006426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cd0f42ce-352e-46f1-8e07-a914e61e3d0c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:35.007611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cd0f42ce-352e-46f1-8e07-a914e61e3d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:35.072170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 69/145] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:35.085497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8b99c-1f3b-4116-96cc-c253fbd15c3f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.085719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8b99c-1f3b-4116-96cc-c253fbd15c3f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.085986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8b99c-1f3b-4116-96cc-c253fbd15c3f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.087441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 77/146] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:35.092287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.093127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.093400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.114906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:35.116417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:35.184749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 70/147] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:35.199782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f8b99c-1f3b-4116-96cc-c253fbd15c3f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.305202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f8b99c-1f3b-4116-96cc-c253fbd15c3f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.339020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.339578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.339857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.340125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.350912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:35.351064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:35.356160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:35.357282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:35.358271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:35.359345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 78/148] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:35.362936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.397533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:35.400006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:35.458498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851fa40-f1d4-4c73-8fd5-ca45ff19a4e3 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.463591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860c221-3504-42f2-9afe-f103a7d564dd None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.492557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.493240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.493521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.493789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.496493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.497118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.497388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.497652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.498298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:35.498592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.500266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.504033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:35.508199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 79/149] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:35.508844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:35.509015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:35.509832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:35.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 71/150] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:35.513754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.607290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:35.612808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c1f9bd-d8b0-423d-8c59-5e6432f3b071 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.613174 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abf4472-471d-4c92-9dad-bd51f17cc3da None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.615084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:35.646256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.646880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.647154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.647418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.647700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.648321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.648506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.648605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.648878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] PATH_INFO: `/v3/users/94160216514f4dc0a8d96135966d5b1c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.650002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.651561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 80/151] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:35.652821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.664106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] RBAC: Authorizing `identity:get_user(user_id=94160216514f4dc0a8d96135966d5b1c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.665656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.671746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 72/152] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/users/94160216514f4dc0a8d96135966d5b1c => generated 283 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:35.671746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.750394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61f22f-7a31-4262-a8a7-9d3f0f9ea204 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.763671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6916069-ccda-4870-8e52-29e3a5157ce7 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.782747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.783345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.783624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.783890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.784537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:35.784865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.786250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.793922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 81/153] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:35.794774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.799780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.800388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.800659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.800954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.801592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.803008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.824082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:35.825937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 73/154] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:35 2022] POST /identity/v3/users => generated 297 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:35.827411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.849066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:35.911663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba200500-436c-4480-8e3f-c096fbc600f1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.944596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b04283-b29f-46a2-a9dd-a4a74d9a7830 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:35.949889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.950629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.950978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.951322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.953271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.954869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.956716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 82/155] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:35.962648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:35.984878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:35.985539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:35.985819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:35.986091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] PATH_INFO: `/v3/users/94160216514f4dc0a8d96135966d5b1c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:35.987580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] RBAC: Authorizing `identity:get_user(user_id=94160216514f4dc0a8d96135966d5b1c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:35.988755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:35.990695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 74/156] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/users/94160216514f4dc0a8d96135966d5b1c => generated 283 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:35.995961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:36.074943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:36.082927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88fcbee-99df-47af-a92a-568ee76032e8 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:36.110221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:36.135240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:36.135916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:36.136194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:36.136460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1abe6a1-17da-4894-ab32-3ea920c7bc82 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:36.136617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9/users/94160216514f4dc0a8d96135966d5b1c/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:36.139713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] RBAC: Authorizing `identity:create_grant(project_id=476619be12f54d90aed177e72373e0d9, user_id=94160216514f4dc0a8d96135966d5b1c, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:36.141316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:36.159021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 83/157] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:35 2022] PUT /identity/v3/projects/476619be12f54d90aed177e72373e0d9/users/94160216514f4dc0a8d96135966d5b1c/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:16:36.190424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203fdbd-50b0-44f4-9dd3-6f59d3d20e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:36.191358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203fdbd-50b0-44f4-9dd3-6f59d3d20e07 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:36.191794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203fdbd-50b0-44f4-9dd3-6f59d3d20e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:36.192214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203fdbd-50b0-44f4-9dd3-6f59d3d20e07 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:36.193975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203fdbd-50b0-44f4-9dd3-6f59d3d20e07 None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:36.195784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203fdbd-50b0-44f4-9dd3-6f59d3d20e07 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:36.197814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 75/158] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:35 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:36.204395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:36.320959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f01563d-2cb4-43e7-9aec-4c89678bea72 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:36.356592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d5be2-d6a7-43e2-9e2f-60963514cf1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:36.357293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d5be2-d6a7-43e2-9e2f-60963514cf1d None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:36.357567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d5be2-d6a7-43e2-9e2f-60963514cf1d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:36.357833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d5be2-d6a7-43e2-9e2f-60963514cf1d None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9/users/94160216514f4dc0a8d96135966d5b1c/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:36.360501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d5be2-d6a7-43e2-9e2f-60963514cf1d None admin] RBAC: Authorizing `identity:create_grant(project_id=476619be12f54d90aed177e72373e0d9, user_id=94160216514f4dc0a8d96135966d5b1c, role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:36.362060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d5be2-d6a7-43e2-9e2f-60963514cf1d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:36.376209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 84/159] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:36 2022] PUT /identity/v3/projects/476619be12f54d90aed177e72373e0d9/users/94160216514f4dc0a8d96135966d5b1c/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:38.131740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70c0035-ba1b-4200-8cd3-6b4142516634 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.132312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70c0035-ba1b-4200-8cd3-6b4142516634 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.132312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70c0035-ba1b-4200-8cd3-6b4142516634 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.134008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 76/160] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:38.140083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fca6a2-3e7d-4b06-878a-676991900578 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.140623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fca6a2-3e7d-4b06-878a-676991900578 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.140623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fca6a2-3e7d-4b06-878a-676991900578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.160943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b4fca6a2-3e7d-4b06-878a-676991900578 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:38.162527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b4fca6a2-3e7d-4b06-878a-676991900578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:38.172652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:38.233799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 85/161] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:38.241587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2136f88-0d4d-428e-80e3-7da26732325c None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.241869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2136f88-0d4d-428e-80e3-7da26732325c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.242142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2136f88-0d4d-428e-80e3-7da26732325c None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.243481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 77/162] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:38.249110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbfeeef-bfc7-4d04-8fcf-dc22efdbb878 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.249399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbfeeef-bfc7-4d04-8fcf-dc22efdbb878 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.249661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbfeeef-bfc7-4d04-8fcf-dc22efdbb878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.267845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1cbfeeef-bfc7-4d04-8fcf-dc22efdbb878 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:38.269069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1cbfeeef-bfc7-4d04-8fcf-dc22efdbb878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:38.347011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 86/163] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:38.416300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ca89c3-83a9-4ce1-ac1a-c04686153d89 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.416956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ca89c3-83a9-4ce1-ac1a-c04686153d89 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.417502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ca89c3-83a9-4ce1-ac1a-c04686153d89 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.419460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 78/164] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:38.426202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff23d0c0-f3ec-4ff0-9722-dec371e239dd None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.426842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff23d0c0-f3ec-4ff0-9722-dec371e239dd None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.427340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff23d0c0-f3ec-4ff0-9722-dec371e239dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.447487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ff23d0c0-f3ec-4ff0-9722-dec371e239dd None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:38.449108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ff23d0c0-f3ec-4ff0-9722-dec371e239dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:38.511419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ca89c3-83a9-4ce1-ac1a-c04686153d89 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:38.522636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 87/165] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:38.544082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6392ee18-d376-41e2-ae34-c7bd2a03116c None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.544380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6392ee18-d376-41e2-ae34-c7bd2a03116c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.544626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6392ee18-d376-41e2-ae34-c7bd2a03116c None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.546507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 88/166] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:38.551441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.551715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.551992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.559103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:38.570473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:38.571746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:38.622821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ca89c3-83a9-4ce1-ac1a-c04686153d89 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:38.652525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 89/167] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:38.653401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:38.661621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:38.662496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.662973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.663465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.675512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:38.681124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:38.682069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:38.682861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:38.685262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 79/168] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:38.691768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:38.712258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:38.794852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be485d9-2b95-4ef8-9d0d-a3d6a95531b6 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:38.799211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32149e7b-5d12-41ce-96a0-21148e7c5d0b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:38.834442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:38.835422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.835796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:38.836149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.836405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.836680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.836983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.837285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.837642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:38.837942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:38.838969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:38.839318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:38.840882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:38.843257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 90/169] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:38.844207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:38.851963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 80/170] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:38.852145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:38.852239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:38.931107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:38.952427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a2cfe6-4282-4637-b95a-46152e65a3e7 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:38.957233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6d366b-e013-49c0-aba5-d6ddf8bfcaf7 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:38.985260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:38.985879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.986152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.986417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.989451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:38.990047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:38.990319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:38.990582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:38.991227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:38.992658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:38.996364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:39.001741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:39.002803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 91/171] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:16:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:39.003696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.017555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 81/172] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:38 2022] POST /identity/v3/users => generated 297 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:39.018377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.078389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:39.128930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430aaaa1-0dbc-4298-a9b2-9b5ff071a3de None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.132818 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8cb5eb-f93b-4803-ae69-4414dc22ab39 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.165838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:39.168407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.169349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.169813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.171052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] PATH_INFO: `/v3/users/ff8df2c383b24828b13da6643613972e` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.172451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.173118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.173769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.174109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.187081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] RBAC: Authorizing `identity:get_user(user_id=ff8df2c383b24828b13da6643613972e)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.187263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:39.188743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.190195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 92/173] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/users/ff8df2c383b24828b13da6643613972e => generated 297 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:39.190961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:39.196964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:39.197861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 82/174] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:39.198622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.315368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c289120-2698-468d-b49a-c71f00d09db1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.322033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9592fc-a753-4cb8-bda9-acd4cadcc571 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.339408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:39.356879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.358398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.358832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.359102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.359764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:39.360066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.362068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.362184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.362598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.362873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.363142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.364943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.366369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.368012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 83/175] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:39.369687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.371040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 93/176] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:39.375334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.444354 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:39.491466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1c96b4-6564-4994-a96a-e13d4e1a33ee None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.509804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0145d27-0d22-4860-99d1-d0e1a8394747 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.528857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.529507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.530324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.530596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.531256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:39.531557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.533057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.544742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 84/177] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:39.544742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.561985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.563589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.564451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.564745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9/users/ff8df2c383b24828b13da6643613972e/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.568543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] RBAC: Authorizing `identity:create_grant(project_id=476619be12f54d90aed177e72373e0d9, user_id=ff8df2c383b24828b13da6643613972e, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.570051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.590982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 94/178] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:39 2022] PUT /identity/v3/projects/476619be12f54d90aed177e72373e0d9/users/ff8df2c383b24828b13da6643613972e/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:39.596881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.694340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:39.745510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a6fe7-4f22-4418-a9af-694e097a754b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.793316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:39.799791 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1194ec6-2725-42a1-b994-9764e6138a96 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:39.821811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.822461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.822737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.823005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] PATH_INFO: `/v3/users/94160216514f4dc0a8d96135966d5b1c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.824746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] RBAC: Authorizing `identity:get_user(user_id=94160216514f4dc0a8d96135966d5b1c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.826377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.828474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 85/179] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/users/94160216514f4dc0a8d96135966d5b1c => generated 283 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:39.835693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:39.852458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:39.853178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:39.853455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:39.853726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] PATH_INFO: `/v3/users/94160216514f4dc0a8d96135966d5b1c` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:39.856449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] RBAC: Authorizing `identity:get_user(user_id=94160216514f4dc0a8d96135966d5b1c)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:39.858797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:39.861773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 95/180] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/users/94160216514f4dc0a8d96135966d5b1c => generated 283 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:39.869947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:40.029346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf77c2a-93d8-4bfa-b888-6a977b4b468d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:40.054130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af5d5a-c9d7-43ad-87c0-6cc8a8dc1d6d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:40.073714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:40.094127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:40.094736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:40.095067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:40.095335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:40.124459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:40.125221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:40.125497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:40.125759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:40.196474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:40.215781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:40.218268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:40.252993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:40.255755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:40.375043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 86/181] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:40.381056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:40.461646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 96/182] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 592 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:40.468261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:40.488048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:40.536848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:40.558552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea506465-7d64-4c9a-a468-38cbc4f5d53c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:40.595795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c339658-1272-46cc-8c97-073c3e0032f1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:40.629840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:40.630399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:40.630670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:40.630933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:40.632420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:40.633897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:40.637145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 87/183] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:40 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:40.642731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:40.662611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:40.663244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:40.663512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:40.663775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:40.665634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:40.667094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:40.668900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 97/184] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:40 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:40.676981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:40.821743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6d3481-d681-4954-8531-6cdc2a561d61 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:40.825853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:40.865435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c681c-9df9-491f-bcfd-dfeeca56bdb3 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:40.886279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:40.886821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:40.887099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:40.887366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:40.888093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=476619be12f54d90aed177e72373e0d9, user.id=94160216514f4dc0a8d96135966d5b1c) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:40.888395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:40.889914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932dd31-cb86-4883-b42e-bf09307e82da None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:40.901410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 88/185] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:40 2022] GET /identity/v3/role_assignments?user.id=94160216514f4dc0a8d96135966d5b1c&scope.project.id=476619be12f54d90aed177e72373e0d9&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:40.922353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:40.922968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:40.923241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:40.923510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:40.924242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] RBAC: Adding query filter params (role.id=cd7b8279c49f4441b98ba176e7e27199, scope.project.id=476619be12f54d90aed177e72373e0d9, user.id=94160216514f4dc0a8d96135966d5b1c) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:40.924545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:40.926036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e95eef7-298f-4a64-93dd-21b1d41875bf None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:40.933311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 98/186] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:40 2022] GET /identity/v3/role_assignments?user.id=94160216514f4dc0a8d96135966d5b1c&scope.project.id=476619be12f54d90aed177e72373e0d9&role.id=cd7b8279c49f4441b98ba176e7e27199 => generated 608 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:41.508251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a051e577-1a2a-4954-b4a2-07177aad1c19 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:41.508792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a051e577-1a2a-4954-b4a2-07177aad1c19 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:41.508792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a051e577-1a2a-4954-b4a2-07177aad1c19 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:41.510296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 89/187] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:41.515828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303c1ec2-ae11-4db1-b7ff-90e2e15699e1 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:41.516261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303c1ec2-ae11-4db1-b7ff-90e2e15699e1 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:41.516411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303c1ec2-ae11-4db1-b7ff-90e2e15699e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:41.535456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-303c1ec2-ae11-4db1-b7ff-90e2e15699e1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:41.536692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-303c1ec2-ae11-4db1-b7ff-90e2e15699e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:41.610240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 99/188] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:41.663729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a2aa6-e933-483e-b253-27233bbed0f1 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:41.664009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a2aa6-e933-483e-b253-27233bbed0f1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:41.664280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a2aa6-e933-483e-b253-27233bbed0f1 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:41.665674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 90/189] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:41.670898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d57215-606e-45cd-a91c-f2b8d2c38962 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:41.671185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d57215-606e-45cd-a91c-f2b8d2c38962 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:41.671464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d57215-606e-45cd-a91c-f2b8d2c38962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:41.689942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-70d57215-606e-45cd-a91c-f2b8d2c38962 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:41.691130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-70d57215-606e-45cd-a91c-f2b8d2c38962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:41.758701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 100/190] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:41.880923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995a2aa6-e933-483e-b253-27233bbed0f1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:41.906533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:41.953566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c1b261-787c-4aaf-93af-6315548ed629 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:41.953854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c1b261-787c-4aaf-93af-6315548ed629 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:41.954127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c1b261-787c-4aaf-93af-6315548ed629 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:41.955605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 101/191] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:41.960251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:41.960530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:41.960828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:41.967023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:41.981537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:41.983047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:41.987112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995a2aa6-e933-483e-b253-27233bbed0f1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.024357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.024938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.025211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.025478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.037336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:42.042111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:42.043009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:42.043766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 91/192] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:42.045002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.055295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 102/193] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:42.057219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.142484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.162566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5063cbbb-fb69-4295-a442-ce0c18e40cdd None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.162706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c206d2-a699-4d86-8ee8-79fbfbdcff54 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.194379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.198000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.198633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.198907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.199182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.200291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.200525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.201254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.202143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.202330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.202539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.203274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:42.203642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.204118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 92/194] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:42.209249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.209778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.218643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 103/195] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:42.224261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.315791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d79326-6db9-40f2-a71f-1fd8a464916b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.331138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83f8f48-e47c-4630-8b98-345b8c00324e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.350937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.351540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.351815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.352087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.352761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.354192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.364589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.365248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.365516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.365778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] PATH_INFO: `/v3/users/c7c42f3b0f4e40ccbfc68504198e816c` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.375095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 93/196] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:42 2022] POST /identity/v3/users => generated 293 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:16:42.375724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.385463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.398853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] RBAC: Authorizing `identity:get_user(user_id=c7c42f3b0f4e40ccbfc68504198e816c)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.400436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.402442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 104/197] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/users/c7c42f3b0f4e40ccbfc68504198e816c => generated 297 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:42.408495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.474889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.493072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3f668-3fc4-4521-acae-2ca0b0a769c2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.527235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a7675c-d881-429f-bb52-72e7f33ca063 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.532598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.533428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.533776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.534115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.546001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:42.551366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:42.552279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:42.553201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:42.556841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 94/198] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:42.558190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.569362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.570028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.570309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.570576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.571737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.573241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.574875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 105/199] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:42.581182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.631565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.676490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d20267-6c8a-4b65-9b8e-b73b3baf3431 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.706536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163097e5-4c80-4cc5-b608-35207b9d82ad None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.722820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.723442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.723721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.723997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.724673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:42.725004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.726398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.734555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.735013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 95/200] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:42.741080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:42.751127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.751799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.752501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.752808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9/users/c7c42f3b0f4e40ccbfc68504198e816c/roles/127abf9263f6445bb39b087269e222f3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.755136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] RBAC: Authorizing `identity:create_grant(project_id=476619be12f54d90aed177e72373e0d9, user_id=c7c42f3b0f4e40ccbfc68504198e816c, role_id=127abf9263f6445bb39b087269e222f3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.757235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.774890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 106/201] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:16:42 2022] PUT /identity/v3/projects/476619be12f54d90aed177e72373e0d9/users/c7c42f3b0f4e40ccbfc68504198e816c/roles/127abf9263f6445bb39b087269e222f3 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:16:42.859814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d235a3-6a5c-4bef-8192-fb4516575a60 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:42.889643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:42.898479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:42.899130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:42.899406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:42.899671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] PATH_INFO: `/v3/users/ff8df2c383b24828b13da6643613972e` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:42.900894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] RBAC: Authorizing `identity:get_user(user_id=ff8df2c383b24828b13da6643613972e)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:42.902363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:42.903993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 96/202] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/users/ff8df2c383b24828b13da6643613972e => generated 297 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:42.910102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:43.019513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db034e35-7d28-4171-b4bc-ce226a0a6fea None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:43.055860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:43.056476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:43.056793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:43.057067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:43.111684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:43.134780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:43.136683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:43.239066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 107/203] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:43.245463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:43.349783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efd9a1-b613-493e-a876-fc5e69f1392c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:43.384563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b77308-3a9a-4b71-ba2e-7e56bf0a3a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:43.385152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b77308-3a9a-4b71-ba2e-7e56bf0a3a6f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:43.385426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b77308-3a9a-4b71-ba2e-7e56bf0a3a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:43.385694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b77308-3a9a-4b71-ba2e-7e56bf0a3a6f None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:43.386821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b77308-3a9a-4b71-ba2e-7e56bf0a3a6f None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:43.388269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b77308-3a9a-4b71-ba2e-7e56bf0a3a6f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:43.394920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 97/204] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:43 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:43.396295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:43.485603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:43.509450 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1016c4f-6115-4155-b63c-e720470dffcb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:43.545752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:43.546381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:43.546633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:43.546900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:43.547623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] RBAC: Adding query filter params (role.id=7914a50fd85645bd870c4f7502de5b02, scope.project.id=476619be12f54d90aed177e72373e0d9, user.id=ff8df2c383b24828b13da6643613972e) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:43.547925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:43.549398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb4b785-542a-4893-ae2b-1eba62e885d8 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:43.555765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 108/205] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:43 2022] GET /identity/v3/role_assignments?user.id=ff8df2c383b24828b13da6643613972e&scope.project.id=476619be12f54d90aed177e72373e0d9&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:44.724533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1115646-3a20-44b3-b37e-3d16c2bddc0b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:44.725085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1115646-3a20-44b3-b37e-3d16c2bddc0b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:44.725085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1115646-3a20-44b3-b37e-3d16c2bddc0b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:44.726435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 98/206] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:44.731631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12346670-8c2a-45a2-b045-a777633c3f4b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:44.732038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12346670-8c2a-45a2-b045-a777633c3f4b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:44.732177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12346670-8c2a-45a2-b045-a777633c3f4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:44.751623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-12346670-8c2a-45a2-b045-a777633c3f4b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:44.752834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-12346670-8c2a-45a2-b045-a777633c3f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:44.775527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134f9d43-cb13-4eef-b94f-1d96e279be88 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:44.775807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134f9d43-cb13-4eef-b94f-1d96e279be88 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:44.776075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134f9d43-cb13-4eef-b94f-1d96e279be88 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:44.777557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 99/207] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:44.782934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0fda45-ba80-4d0a-9c61-ccefe78d22b3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:44.783215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0fda45-ba80-4d0a-9c61-ccefe78d22b3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:44.783490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0fda45-ba80-4d0a-9c61-ccefe78d22b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:44.801782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0d0fda45-ba80-4d0a-9c61-ccefe78d22b3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:44.802972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0d0fda45-ba80-4d0a-9c61-ccefe78d22b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:44.822121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 109/208] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:44.865629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:44.875111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 100/209] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:44.997985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78dfb42-e6ff-49c4-b099-c3d202a83f5a None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:44.999070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78dfb42-e6ff-49c4-b099-c3d202a83f5a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:44.999251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78dfb42-e6ff-49c4-b099-c3d202a83f5a None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.000755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 110/210] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:45.005850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.006130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.006402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.024381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:45.025565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:45.090238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 101/211] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:45.091547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78dfb42-e6ff-49c4-b099-c3d202a83f5a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.148963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.169599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.200961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78dfb42-e6ff-49c4-b099-c3d202a83f5a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.237076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.237698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.237971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.238235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.248297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:45.253342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:45.254300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:45.255227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 111/212] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:45.260029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.261845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b843c5-0d96-42e3-ab5e-f2c7998feb0a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.287086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.299182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.299803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.300074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.300336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.301445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.302869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.304755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 102/213] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:45.309826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.365013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf16c32-4602-4dc9-b571-8356dda2f8e7 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.397817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.398424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.398695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.398961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.399612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] RBAC: Adding query filter params (name=member) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:45.399914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.401330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.402625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.409647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b633df0-c2ea-445e-a5f4-8e2a415765ca None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.411543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 112/214] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:45.412349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.442536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.443133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.443406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.443667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.444306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.445769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.466376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 103/215] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:16:45 2022] POST /identity/v3/users => generated 293 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:45.467677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.518095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2f3328-014b-47e6-bae6-f298f24e1b01 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.530337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.556818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.557473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.557746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.558009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.570129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:45.575552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:45.576531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:45.577855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:45.582344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 113/216] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:45.583321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.594703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54abd7f-c7c0-42fa-b165-47fe5c5a1a3f None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.635687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.636752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.637229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.637672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] PATH_INFO: `/v3/users/8d19104d35684c30839745298548e4c7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.654255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.654440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] RBAC: Authorizing `identity:get_user(user_id=8d19104d35684c30839745298548e4c7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.656047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.657872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 104/217] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/users/8d19104d35684c30839745298548e4c7 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:45.663623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.705589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419a8920-74d6-4ae9-ba1a-78131764d46a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.752067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.752871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.753270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.753535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.754209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:45.754678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.756607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.770227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 114/218] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:45.775442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac7d3ad-4040-406a-8909-d94e7ed87813 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.775908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:45.784919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.824449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.825125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.825696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.825972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.827492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] RBAC: Adding query filter params (role.id=7914a50fd85645bd870c4f7502de5b02, user.id=8d19104d35684c30839745298548e4c7, scope.system=all) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:45.827807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.830271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.835326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 105/219] 10.222.0.23 () {56 vars in 1308 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/role_assignments?user.id=8d19104d35684c30839745298548e4c7&scope.system=all&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:45.897285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f738da74-6dc9-40ea-afa4-04af2fd1da4b None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:45.918266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:45.940044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:45.941093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:45.941573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:45.942070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] PATH_INFO: `/v3/users/c7c42f3b0f4e40ccbfc68504198e816c` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:45.943555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] RBAC: Authorizing `identity:get_user(user_id=c7c42f3b0f4e40ccbfc68504198e816c)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:45.945406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:45.947857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 115/220] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/users/c7c42f3b0f4e40ccbfc68504198e816c => generated 297 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:45.953139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:46.064965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebc0d13-7424-411b-9d7d-b001878493a6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:46.105538 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:46.106219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:46.106493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:46.106760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:46.153130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:46.183925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:46.185414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:46.287938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 106/221] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:16:46.293707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:46.403647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9fb224-8704-4d62-b326-91d830af8e47 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:46.438943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b075a2-4f15-4b12-97d5-6b239aa88f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:46.439578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b075a2-4f15-4b12-97d5-6b239aa88f28 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:46.439851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b075a2-4f15-4b12-97d5-6b239aa88f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:46.440118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b075a2-4f15-4b12-97d5-6b239aa88f28 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:46.441318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b075a2-4f15-4b12-97d5-6b239aa88f28 None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:46.442834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b075a2-4f15-4b12-97d5-6b239aa88f28 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:46.444466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 116/222] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:46 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:46.450382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:46.521502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:46.555057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd48400-fab7-40b7-ae29-cda9a1266fae None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:46.589586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:46.590202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:46.590474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:46.590744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:46.591485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] RBAC: Adding query filter params (role.id=127abf9263f6445bb39b087269e222f3, scope.project.id=476619be12f54d90aed177e72373e0d9, user.id=c7c42f3b0f4e40ccbfc68504198e816c) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:46.591792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:46.593367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb2fe8c-effb-4371-82c2-7002b90a4187 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:46.600171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 107/223] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:16:46 2022] GET /identity/v3/role_assignments?user.id=c7c42f3b0f4e40ccbfc68504198e816c&scope.project.id=476619be12f54d90aed177e72373e0d9&role.id=127abf9263f6445bb39b087269e222f3 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:47.811467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256566cb-2d11-4d3c-af17-e2bbaf390d48 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:47.812007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256566cb-2d11-4d3c-af17-e2bbaf390d48 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:47.812007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256566cb-2d11-4d3c-af17-e2bbaf390d48 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:47.813561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 117/224] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:47.818533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7182e83b-f41d-4aa8-a70c-eb60fa7c5f0b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:47.818942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7182e83b-f41d-4aa8-a70c-eb60fa7c5f0b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:47.819129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7182e83b-f41d-4aa8-a70c-eb60fa7c5f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:47.837368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-7182e83b-f41d-4aa8-a70c-eb60fa7c5f0b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:47.838545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7182e83b-f41d-4aa8-a70c-eb60fa7c5f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:47.885402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3f9fa9-39ae-453e-b3cb-e7b857890f01 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:47.885696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3f9fa9-39ae-453e-b3cb-e7b857890f01 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:47.885944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3f9fa9-39ae-453e-b3cb-e7b857890f01 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:47.887304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 118/225] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:47.892323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f5ed6d-9c48-4ba9-87bb-da5a08468492 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:47.892603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f5ed6d-9c48-4ba9-87bb-da5a08468492 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:47.892891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f5ed6d-9c48-4ba9-87bb-da5a08468492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:47.907184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 108/226] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:47.913377 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-35f5ed6d-9c48-4ba9-87bb-da5a08468492 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:47.914563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-35f5ed6d-9c48-4ba9-87bb-da5a08468492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:47.966524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:47.987589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 119/227] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:48.072010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f1fa96-5fed-4c8f-b8ec-6e99838eef4f None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.072240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f1fa96-5fed-4c8f-b8ec-6e99838eef4f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.072672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f1fa96-5fed-4c8f-b8ec-6e99838eef4f None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.074154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 109/228] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:48.079775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.080057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.080324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.100050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:48.101260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:48.173046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 120/229] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:48.175084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f1fa96-5fed-4c8f-b8ec-6e99838eef4f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.239578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:48.255153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.285278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f1fa96-5fed-4c8f-b8ec-6e99838eef4f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.322315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.322869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.323143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.323417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.335387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:48.340942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:48.341803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:48.342754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 110/230] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:48.347798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.362139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7f6ec7-1e99-4b83-aa2b-3b3d168c1055 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.378775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:48.401212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.402060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.402596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.402870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.403980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:48.406349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:48.412490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 121/231] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:48.413303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.450503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83382003-96b8-40af-acb8-eebe0f890759 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.473504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:48.484705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.485328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.485597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.485859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.486507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:48.486797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:48.488165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:48.495992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 111/232] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:48.496797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.518866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe178f3-a11d-4c96-8efd-6ba84cb162b2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.553029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.553633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.553906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.554176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] PATH_INFO: `/v3/groups` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.554798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:48.556248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:48.571918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 122/233] 10.222.0.23 () {60 vars in 1134 bytes} [Wed Oct 12 02:16:48 2022] POST /identity/v3/groups => generated 229 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:48.572801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.615364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4bf741-b07b-4baf-91af-90fbfee10794 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.620157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:48.657833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.658487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.658766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.659038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.672646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:48.678328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:48.679281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:48.680101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 112/234] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:48.685476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.691787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120d06d-3584-48f2-9766-93a6ea052fb5 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.736293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.737037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.737803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.737803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] PATH_INFO: `/v3/users/0ce0a18aa00a4c00aa0f1e31afca3a96` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.745683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:48.753099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] RBAC: Authorizing `identity:get_user(user_id=0ce0a18aa00a4c00aa0f1e31afca3a96)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:48.754638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:48.756628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 123/235] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/users/0ce0a18aa00a4c00aa0f1e31afca3a96 => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:48.762097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.805689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e53c914-21cd-40bb-b561-3a5c8393fa1a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.842517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.843139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.843363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.843629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.844415 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:48.844735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:48.846241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:48.856409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 113/236] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:48.862324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:48.864617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:48.869308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633bb216-c0cd-46c9-a109-9db69f4a5d5d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.909504 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:48.910334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:48.910442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:48.910637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:48.911355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] RBAC: Adding query filter params (role.id=127abf9263f6445bb39b087269e222f3, user.id=0ce0a18aa00a4c00aa0f1e31afca3a96, scope.system=all) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:48.911658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:48.913137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:48.918283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 124/237] 10.222.0.23 () {56 vars in 1308 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/role_assignments?user.id=0ce0a18aa00a4c00aa0f1e31afca3a96&scope.system=all&role.id=127abf9263f6445bb39b087269e222f3 => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:48.979526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e4f66-2ab8-4273-bdcf-1026d4b9779e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:48.989969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:49.018495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d251a5-1f5d-4235-b221-f85c986e8132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:49.019152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d251a5-1f5d-4235-b221-f85c986e8132 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:49.019419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d251a5-1f5d-4235-b221-f85c986e8132 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:49.019687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d251a5-1f5d-4235-b221-f85c986e8132 None admin] PATH_INFO: `/v3/users/8d19104d35684c30839745298548e4c7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:49.020898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d251a5-1f5d-4235-b221-f85c986e8132 None admin] RBAC: Authorizing `identity:get_user(user_id=8d19104d35684c30839745298548e4c7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:49.022375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d251a5-1f5d-4235-b221-f85c986e8132 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:49.024248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 114/238] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:48 2022] GET /identity/v3/users/8d19104d35684c30839745298548e4c7 => generated 293 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:49.029802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:49.141491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bf903e-3717-4ee3-a1e9-a9f40c4b3915 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:49.175903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6344331-771a-4c99-a503-5cb384540b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:49.176559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6344331-771a-4c99-a503-5cb384540b2f None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:49.176863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6344331-771a-4c99-a503-5cb384540b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:49.177135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6344331-771a-4c99-a503-5cb384540b2f None admin] PATH_INFO: `/v3/system/users/8d19104d35684c30839745298548e4c7/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:49.178533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6344331-771a-4c99-a503-5cb384540b2f None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=8d19104d35684c30839745298548e4c7, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:49.179922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6344331-771a-4c99-a503-5cb384540b2f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:49.189597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 125/239] 10.222.0.23 () {58 vars in 1334 bytes} [Wed Oct 12 02:16:49 2022] PUT /identity/v3/system/users/8d19104d35684c30839745298548e4c7/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:50.765671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501ea100-b7d4-4e30-9a52-b12e53ca588b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:50.766182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501ea100-b7d4-4e30-9a52-b12e53ca588b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:50.766256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501ea100-b7d4-4e30-9a52-b12e53ca588b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:50.767715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 115/240] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:50.772633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705c331-d422-482b-8a48-edf67ae41fab None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:50.773028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705c331-d422-482b-8a48-edf67ae41fab None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:50.773222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705c331-d422-482b-8a48-edf67ae41fab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:50.791842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5705c331-d422-482b-8a48-edf67ae41fab None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:50.793056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5705c331-d422-482b-8a48-edf67ae41fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:50.816551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:50.862523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 126/241] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:51.062434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404c2e5b-9814-4ec8-b714-446c6c943b0d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.062783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404c2e5b-9814-4ec8-b714-446c6c943b0d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.062896 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404c2e5b-9814-4ec8-b714-446c6c943b0d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.064411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 116/242] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:51.069339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.069645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.069890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.088219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:51.089422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:51.127385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404c2e5b-9814-4ec8-b714-446c6c943b0d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.160743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 127/243] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:51.236048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404c2e5b-9814-4ec8-b714-446c6c943b0d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:51.269771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69290315-307a-4e50-b4e0-8d85bf3fa07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:51.270398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69290315-307a-4e50-b4e0-8d85bf3fa07e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.270674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69290315-307a-4e50-b4e0-8d85bf3fa07e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.270941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69290315-307a-4e50-b4e0-8d85bf3fa07e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.272034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69290315-307a-4e50-b4e0-8d85bf3fa07e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:51.273497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69290315-307a-4e50-b4e0-8d85bf3fa07e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:51.275282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 117/244] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:16:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:51.280600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.310601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2acb1b-6e7c-4732-8f07-219aaa4b60cd None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.311037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2acb1b-6e7c-4732-8f07-219aaa4b60cd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.311306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2acb1b-6e7c-4732-8f07-219aaa4b60cd None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.312675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 118/245] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:51.317454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.317728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.317993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.336629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:51.337858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:51.380760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:51.381056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:51.395353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ad72e6-509a-47da-839a-56efddf21514 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:51.413763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 119/246] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:51.427422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.431891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:51.432775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.433249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.433715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] PATH_INFO: `/v3/groups` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.434563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:51.437366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:51.456199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 128/247] 10.222.0.23 () {60 vars in 1134 bytes} [Wed Oct 12 02:16:51 2022] POST /identity/v3/groups => generated 226 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:16:51.538428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5304319e-a889-40db-9d08-07e4b8c5cafd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:51.575572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:51.576134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.576473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.576769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.588364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:51.594046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:51.594997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:51.595828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:51.600176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 120/248] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:51.601260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.682473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.717878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0c5f0-b736-4982-a742-6021faea64af None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:51.728271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:51.762007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:51.762734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.763087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.763354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.764072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:51.764369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:51.765844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:51.774653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 129/249] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:51.780305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.793920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:51.817928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fa350c-f083-472d-adc8-9399d40d9052 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:51.874886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:51.874886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.874886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.874886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.889333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:51.895249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:51.896135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:51.897371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:51.900924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 121/250] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:51.902914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:51.911378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c2f36d-c9b7-4c06-bda2-0d99d04a43e8 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:51.955403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:51.956047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:51.956346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:51.956587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] PATH_INFO: `/v3/users/0ce0a18aa00a4c00aa0f1e31afca3a96` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:51.957807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] RBAC: Authorizing `identity:get_user(user_id=0ce0a18aa00a4c00aa0f1e31afca3a96)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:51.959270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:51.964728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 130/251] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:51 2022] GET /identity/v3/users/0ce0a18aa00a4c00aa0f1e31afca3a96 => generated 293 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:51.967451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:52.028801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:52.039240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cbba10-950d-47ee-81d2-f89a7003b26d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:52.077096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:52.077785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:52.078058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:52.078321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:52.079019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:52.079317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:52.080718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:52.081843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:52.094029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 122/252] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:52.100888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:52.131411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cccc08d-9eeb-405a-84c8-61d5c6cb955e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:52.185433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:52.186641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:52.186920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:52.187193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] PATH_INFO: `/v3/system/users/0ce0a18aa00a4c00aa0f1e31afca3a96/roles/127abf9263f6445bb39b087269e222f3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:52.188945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=0ce0a18aa00a4c00aa0f1e31afca3a96, role_id=127abf9263f6445bb39b087269e222f3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:52.191145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:52.207352 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 131/253] 10.222.0.23 () {58 vars in 1334 bytes} [Wed Oct 12 02:16:51 2022] PUT /identity/v3/system/users/0ce0a18aa00a4c00aa0f1e31afca3a96/roles/127abf9263f6445bb39b087269e222f3 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:52.242571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c356855e-fed0-4765-a642-62b5dedc1971 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:52.285106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a39e267-9147-4bf8-87c8-049b5cee683a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:52.285824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a39e267-9147-4bf8-87c8-049b5cee683a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:52.286107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a39e267-9147-4bf8-87c8-049b5cee683a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:52.286376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a39e267-9147-4bf8-87c8-049b5cee683a None admin] PATH_INFO: `/v3/users/8d19104d35684c30839745298548e4c7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:52.287742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a39e267-9147-4bf8-87c8-049b5cee683a None admin] RBAC: Authorizing `identity:get_user(user_id=8d19104d35684c30839745298548e4c7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:52.289340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a39e267-9147-4bf8-87c8-049b5cee683a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:52.291381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 123/254] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:52 2022] GET /identity/v3/users/8d19104d35684c30839745298548e4c7 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:52.297571 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:52.419088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee3951-80f9-4ad1-8fdd-639fc77720e7 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:52.455759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:52.456382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:52.456660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:52.456963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:52.457769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] RBAC: Adding query filter params (role.id=7914a50fd85645bd870c4f7502de5b02, user.id=8d19104d35684c30839745298548e4c7, scope.system=all) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:52.458076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:52.459547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaece85f-e06e-4927-810a-68dde1da2e8c None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:52.471346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:52.477044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 132/255] 10.222.0.23 () {56 vars in 1308 bytes} [Wed Oct 12 02:16:52 2022] GET /identity/v3/role_assignments?user.id=8d19104d35684c30839745298548e4c7&scope.system=all&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 510 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:53.833713 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7bf47e-8d7f-4eaf-b356-d4e1db0084b6 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:53.834257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7bf47e-8d7f-4eaf-b356-d4e1db0084b6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:53.834257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7bf47e-8d7f-4eaf-b356-d4e1db0084b6 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:53.835933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 124/256] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:53.842973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc863063-f07b-4703-bf55-418bd88584a7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:53.843400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc863063-f07b-4703-bf55-418bd88584a7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:53.843579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc863063-f07b-4703-bf55-418bd88584a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:53.869801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-fc863063-f07b-4703-bf55-418bd88584a7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:53.871028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-fc863063-f07b-4703-bf55-418bd88584a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:53.939290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 133/257] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:53 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 12 02:16:54.155056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9dc510-1c6d-48fa-b84b-1c87f40f9cd4 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.155392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9dc510-1c6d-48fa-b84b-1c87f40f9cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.155603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9dc510-1c6d-48fa-b84b-1c87f40f9cd4 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.157253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 125/258] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:54.162385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.162665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.162930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.184347 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:54.185594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:54.199792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 134/259] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:54 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 12 02:16:54.505149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9dc510-1c6d-48fa-b84b-1c87f40f9cd4 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.518677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9dc510-1c6d-48fa-b84b-1c87f40f9cd4 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.528603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.529773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.530046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.530311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.546329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:54.551064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:54.551925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:54.552761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:54.553374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 126/260] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 48 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:54.557832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.568031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c10064c-b081-4ce4-abc3-17ce5d8027d9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.575599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.576274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.576546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.576846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.577513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] RBAC: Adding query filter params (name=member) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:54.577805 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:54.579595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:16:54.579595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:16:54.579595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:16:54.580516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:54.589604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 135/261] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:54.595677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.609830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cd487-cc80-4da0-9379-2a5dcc866157 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.614200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.614824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.615098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.615362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.625922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:54.627730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:16:54.627730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:16:54.627730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:16:54.628979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:54.630975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 127/262] 10.222.0.23 () {56 vars in 1201 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:54.636128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.646412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79782bda-e008-4682-b83c-422b8bf5cf7c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.651826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.652511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.652815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.653087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.656235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:54.658423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:54.660064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 136/263] 10.222.0.23 () {56 vars in 1207 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:54.665803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.678109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:54.683473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1618e55-2b13-4a3e-8d46-b2964d8ce333 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.690541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.691217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.691490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.691759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd/groups/a28004e8a06845068d4ab297c642dcb9/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.693665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] RBAC: Authorizing `identity:create_grant(project_id=239dfc9cbb164dc89983358a6d8450bd, group_id=a28004e8a06845068d4ab297c642dcb9, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:54.694676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:16:54.694676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:16:54.694676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:16:54.696766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:54.724728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 128/264] 10.222.0.23 () {58 vars in 1463 bytes} [Wed Oct 12 02:16:54 2022] PUT /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd/groups/a28004e8a06845068d4ab297c642dcb9/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:16:54.782418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.798009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5925a348-52a6-48d9-afc2-6c35fa6767ff admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.873569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.874610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.874775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.874999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.893696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:54.899402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:54.901673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:54.903213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 137/265] 10.222.0.23 () {56 vars in 1135 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 119 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:54.910760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.926659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fdf96c-e73d-4f40-a809-9377a1abf570 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.936813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.937454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.937725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.937988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.938660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:54.938956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:54.939966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:16:54.939966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:16:54.939966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:16:54.940887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:54.958155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 129/266] 10.222.0.23 () {56 vars in 1132 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:54.964097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:54.979089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845edd14-f623-41f6-bbe2-6f27b15ab8fb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:54.988801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:54.989636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:54.989817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:54.989972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:54.992850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:54.993954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:16:54.993954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:16:54.993954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:16:54.995272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:54.997233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 138/267] 10.222.0.23 () {56 vars in 1201 bytes} [Wed Oct 12 02:16:54 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:55.002959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:55.018345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157de098-4a0a-4a8c-a6af-75e05180c705 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:55.026197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931999ff-5de5-472e-a0c3-d9e3e8379915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:55.026835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931999ff-5de5-472e-a0c3-d9e3e8379915 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:55.027109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931999ff-5de5-472e-a0c3-d9e3e8379915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:55.027373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931999ff-5de5-472e-a0c3-d9e3e8379915 admin admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:55.028494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931999ff-5de5-472e-a0c3-d9e3e8379915 admin admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:55.030654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931999ff-5de5-472e-a0c3-d9e3e8379915 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:55.032272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 130/268] 10.222.0.23 () {56 vars in 1207 bytes} [Wed Oct 12 02:16:55 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:55.038351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:55.050948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a757c09a-726f-4372-8629-37829bec2dff admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:55.060888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:55.061556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:55.061833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:55.062102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd/groups/a28004e8a06845068d4ab297c642dcb9/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:55.066516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=239dfc9cbb164dc89983358a6d8450bd, group_id=a28004e8a06845068d4ab297c642dcb9, role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:55.067555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:16:55.067555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:16:55.067555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:16:55.069846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa99657-aea3-40cb-a8d7-f64b0cfdcf28 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:55.100748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 139/269] 10.222.0.23 () {58 vars in 1463 bytes} [Wed Oct 12 02:16:55 2022] PUT /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd/groups/a28004e8a06845068d4ab297c642dcb9/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:55.282223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4d7a-5447-4b0a-b048-2d4863cf066e None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:55.282552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4d7a-5447-4b0a-b048-2d4863cf066e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:55.282747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4d7a-5447-4b0a-b048-2d4863cf066e None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:55.284385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 131/270] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:55.291066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63825f3-ab63-4234-88e0-ba6ed67ea750 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:55.291344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63825f3-ab63-4234-88e0-ba6ed67ea750 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:55.291521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63825f3-ab63-4234-88e0-ba6ed67ea750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:55.315184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f63825f3-ab63-4234-88e0-ba6ed67ea750 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:55.316415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f63825f3-ab63-4234-88e0-ba6ed67ea750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:55.386222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:55.411622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 140/271] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:16:55.528689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52698e91-2dc6-4fac-ae5c-bc89839fafe2 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:55.529063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52698e91-2dc6-4fac-ae5c-bc89839fafe2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:55.529254 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52698e91-2dc6-4fac-ae5c-bc89839fafe2 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:55.530723 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 132/272] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:55.539444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:55.539444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:55.539444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:55.561534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:55.563095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:55.645159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 141/273] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:55 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 12 02:16:55.912379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52698e91-2dc6-4fac-ae5c-bc89839fafe2 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.051436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52698e91-2dc6-4fac-ae5c-bc89839fafe2 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.107120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21f79ba-df1a-4a52-bb30-74ab4f80047a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.108197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21f79ba-df1a-4a52-bb30-74ab4f80047a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.108720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21f79ba-df1a-4a52-bb30-74ab4f80047a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.109217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21f79ba-df1a-4a52-bb30-74ab4f80047a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.129398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21f79ba-df1a-4a52-bb30-74ab4f80047a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c21f79ba-df1a-4a52-bb30-74ab4f80047a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:56.135164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:56.136029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 12 02:16:56.136906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:56.139588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 133/274] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:56.145172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.174921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b69ec1f-0d40-43e9-b7ff-bb1e117bb16f None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.175515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b69ec1f-0d40-43e9-b7ff-bb1e117bb16f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.175986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b69ec1f-0d40-43e9-b7ff-bb1e117bb16f None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.177551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 134/275] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:56.193575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.194138 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.194588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.237214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:56.238440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:56.258971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 135/276] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:56 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 12 02:16:56.285707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.299269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6fd94b-88f0-446c-97aa-34e0928429f5 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.305984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.306827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.307106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.307374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.312690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111ffc9-0d60-4c59-b40e-ba790da23f02 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.335079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:56.341023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:56.341971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:56.342770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 136/277] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 57 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:56.351677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.355951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:56.373143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc5aaf-1b08-44a3-b363-f3fa62cdbfc0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.374707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.375388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.375716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.376097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.376900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:56.377203 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.378676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.383525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.384150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.384426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.384691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.387639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:56.387952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.391260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.394876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 142/278] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.400490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.407469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 137/279] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.413188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.429296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64313f4c-5c1a-4972-8887-232f8470b61c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.440689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.441365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.441642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.442005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.443196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.445341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.447527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 138/280] 10.222.0.23 () {56 vars in 1201 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.453416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.463555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d218cf2-8316-446b-8717-af8ab1b8bd3c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.470302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.471131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.471505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.476722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.476722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.476722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.481198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 139/281] 10.222.0.23 () {56 vars in 1207 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.486550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.498982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:56.504828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f933d92-8fc4-40c8-8833-4509a33b8b37 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.521920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.522585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.522854 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.523125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9/groups/a28004e8a06845068d4ab297c642dcb9/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.533553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] RBAC: Authorizing `identity:create_grant(project_id=476619be12f54d90aed177e72373e0d9, group_id=a28004e8a06845068d4ab297c642dcb9, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.536244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.567630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d73e7-8281-4ab0-bb60-8133e9eb2ecf None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.579136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 140/282] 10.222.0.23 () {58 vars in 1463 bytes} [Wed Oct 12 02:16:56 2022] PUT /identity/v3/projects/476619be12f54d90aed177e72373e0d9/groups/a28004e8a06845068d4ab297c642dcb9/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:56.622584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.623314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.623908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.624188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] PATH_INFO: `/v3/users/0ce0a18aa00a4c00aa0f1e31afca3a96` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.625535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] RBAC: Authorizing `identity:get_user(user_id=0ce0a18aa00a4c00aa0f1e31afca3a96)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.627127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.630627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 143/283] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/users/0ce0a18aa00a4c00aa0f1e31afca3a96 => generated 293 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.636061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.700463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.810263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368fbede-3f2d-447e-b9fe-c9c4c16916ff None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.815383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.816050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.816326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.816596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.819720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:56.835190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654c6431-ebd4-4ad1-b9a5-c22e3965689b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.837923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:56.845246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:56.846184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:56.846957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 144/284] 10.222.0.23 () {56 vars in 1135 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:56.852073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.864338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0bcc29-10bd-4d22-a734-f9f075d16e45 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.873545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.874510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.874787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.875051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.875786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:56.876089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.877804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.893277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 145/285] 10.222.0.23 () {56 vars in 1132 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.898554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.919991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f1fff7-1167-4943-8a5b-04cc0d6fb4c8 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.934480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.935174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.935964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.936209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.936954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] RBAC: Adding query filter params (role.id=127abf9263f6445bb39b087269e222f3, user.id=0ce0a18aa00a4c00aa0f1e31afca3a96, scope.system=all) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:56.937261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.937445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.938098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.938365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.938626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.938725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.939838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.941803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.944483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 146/286] 10.222.0.23 () {56 vars in 1201 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:16:56.948996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:56.962474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849d58d1-bf90-4bbb-ab51-073941c052f6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:56.964407 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 141/287] 10.222.0.23 () {56 vars in 1308 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/role_assignments?user.id=0ce0a18aa00a4c00aa0f1e31afca3a96&scope.system=all&role.id=127abf9263f6445bb39b087269e222f3 => generated 510 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.974823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac203ffb-21de-463c-936f-20cde47b6e5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:56.975488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac203ffb-21de-463c-936f-20cde47b6e5b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:56.975760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac203ffb-21de-463c-936f-20cde47b6e5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:56.976025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac203ffb-21de-463c-936f-20cde47b6e5b admin admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:56.977432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac203ffb-21de-463c-936f-20cde47b6e5b admin admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:56.979507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac203ffb-21de-463c-936f-20cde47b6e5b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:56.985620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 147/288] 10.222.0.23 () {56 vars in 1207 bytes} [Wed Oct 12 02:16:56 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:56.991748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:57.006910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c88ede-89be-4012-b667-3f88c2a43547 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.007233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c88ede-89be-4012-b667-3f88c2a43547 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.007443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c88ede-89be-4012-b667-3f88c2a43547 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.009147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 148/289] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:16:57.014785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.015082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.015321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.025841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e5f7e6-cf39-46cc-995a-f1af259963c2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:57.034440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:57.035120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.035395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.035665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9/groups/a28004e8a06845068d4ab297c642dcb9/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.042618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] RBAC: Authorizing `identity:create_grant(project_id=476619be12f54d90aed177e72373e0d9, group_id=a28004e8a06845068d4ab297c642dcb9, role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:57.045857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:57.053936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:57.055161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:57.072053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 142/290] 10.222.0.23 () {58 vars in 1463 bytes} [Wed Oct 12 02:16:56 2022] PUT /identity/v3/projects/476619be12f54d90aed177e72373e0d9/groups/a28004e8a06845068d4ab297c642dcb9/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:57.082265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 149/291] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:57 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 12 02:16:57.524221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:57.584311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:57.607389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e2361a-43ff-4a17-b48b-3073f5337fba admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:57.612744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:57.613606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.614074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.614532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.626734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:57.636436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:57.637316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:16:57.638106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:16:57.640452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 143/292] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:16:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 117 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:57.649792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:57.661803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dd8149-36f4-456b-a4e1-1e67eb7e5734 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:57.674328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:57.674958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.675231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.675499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.676165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:57.676465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:57.678141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:57.699179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 150/293] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:16:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:57.705253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:57.722322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a728aed-a025-4abd-9a2c-0bcd81173536 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:57.729906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:57.730789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.731256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.731712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] PATH_INFO: `/v3/groups/5418044205d548299c107e00d7c5d070` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.737539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] RBAC: Authorizing `identity:get_group(group_id=5418044205d548299c107e00d7c5d070)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:57.739789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:57.741726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 144/294] 10.222.0.23 () {56 vars in 1201 bytes} [Wed Oct 12 02:16:57 2022] GET /identity/v3/groups/5418044205d548299c107e00d7c5d070 => generated 229 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:57.747434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:57.761831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbd7377-d5c2-447a-b43b-ba31a8d190d1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:57.772461 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae221a8-2828-4c73-a8a6-9371a093f2d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:57.773145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae221a8-2828-4c73-a8a6-9371a093f2d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.773420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae221a8-2828-4c73-a8a6-9371a093f2d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.773687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae221a8-2828-4c73-a8a6-9371a093f2d3 admin admin] PATH_INFO: `/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.777451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae221a8-2828-4c73-a8a6-9371a093f2d3 admin admin] RBAC: Authorizing `identity:get_project(project_id=1b152cd4fded41fa8ba95182a5dc98e4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:57.779321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae221a8-2828-4c73-a8a6-9371a093f2d3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:57.781350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 151/295] 10.222.0.23 () {56 vars in 1207 bytes} [Wed Oct 12 02:16:57 2022] GET /identity/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4 => generated 344 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:57.787000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:57.798235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537dd24-b535-4dea-ba20-2c0e021465ec admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:57.806393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db635a01-fdc3-44dd-994f-51a8521c5ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:57.807336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db635a01-fdc3-44dd-994f-51a8521c5ca2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:57.807883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db635a01-fdc3-44dd-994f-51a8521c5ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:57.808343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db635a01-fdc3-44dd-994f-51a8521c5ca2 admin admin] PATH_INFO: `/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4/groups/5418044205d548299c107e00d7c5d070/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:57.810683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db635a01-fdc3-44dd-994f-51a8521c5ca2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b152cd4fded41fa8ba95182a5dc98e4, group_id=5418044205d548299c107e00d7c5d070, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:57.813787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db635a01-fdc3-44dd-994f-51a8521c5ca2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:57.836472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 145/296] 10.222.0.23 () {58 vars in 1463 bytes} [Wed Oct 12 02:16:57 2022] PUT /identity/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4/groups/5418044205d548299c107e00d7c5d070/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:16:58.293187 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8805f8f-9704-42c3-9aab-e3173843f8fb None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:58.293546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8805f8f-9704-42c3-9aab-e3173843f8fb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:58.293744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8805f8f-9704-42c3-9aab-e3173843f8fb None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:58.295396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 152/297] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:58.306242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc351bb-f712-4171-8494-7c6dafa4b509 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:58.307085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc351bb-f712-4171-8494-7c6dafa4b509 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:58.307549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc351bb-f712-4171-8494-7c6dafa4b509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:58.333102 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0dc351bb-f712-4171-8494-7c6dafa4b509 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:58.334669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0dc351bb-f712-4171-8494-7c6dafa4b509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:58.426312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 146/298] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:58.602232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511d937-78b1-4df0-a1e2-0c8becf9bc0d None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:58.602573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511d937-78b1-4df0-a1e2-0c8becf9bc0d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:58.602829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511d937-78b1-4df0-a1e2-0c8becf9bc0d None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:58.604432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 153/299] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:58.610142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:58.610421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:58.610689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:58.633712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:58.635224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:58.718883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 147/300] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:58.946901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4511d937-78b1-4df0-a1e2-0c8becf9bc0d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.083508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4511d937-78b1-4df0-a1e2-0c8becf9bc0d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:59.128403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:59.129099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.129376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.129646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.145543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:59.151169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:59.152032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:59.152863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 154/301] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:16:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:16:59.160296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.187441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:59.286443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.288177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9163e6a-bc3d-463b-abd8-f3e59d5f77cc None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:59.325713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:59.333915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:59.335211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.335681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.336156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.337063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:59.337761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:59.339342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:59.355313 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 148/302] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:16:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:59.366621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.493111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68b6f4-044d-4cf2-95a4-ae6b5047229d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:59.515872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:59.556432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd579db-5f47-4822-bf6c-d66a81eed45e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:59.561202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa1e329-537d-4257-acfe-2a874d9edfcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:59.561841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa1e329-537d-4257-acfe-2a874d9edfcc None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.562110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa1e329-537d-4257-acfe-2a874d9edfcc None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.562373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa1e329-537d-4257-acfe-2a874d9edfcc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.574144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa1e329-537d-4257-acfe-2a874d9edfcc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-0aa1e329-537d-4257-acfe-2a874d9edfcc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:16:59.581271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:16:59.582178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:16:59.582998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 155/303] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:16:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:16:59.585357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d5242a-e1aa-4cfa-8628-aa8f0d1decd3 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.585575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d5242a-e1aa-4cfa-8628-aa8f0d1decd3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.585841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d5242a-e1aa-4cfa-8628-aa8f0d1decd3 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.587297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 156/304] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:16:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:16:59.589543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d5242a-e1aa-4cfa-8628-aa8f0d1decd3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.609586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:59.610264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.610543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.610806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.694044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:59.706862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:59.709186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:59.726945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d5242a-e1aa-4cfa-8628-aa8f0d1decd3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:59.773308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:16:59.775013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.775500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.776263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.777198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] RBAC: Adding query filter params (name=member) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:16:59.777967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:16:59.781039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badcd3a9-d1c1-49d1-80c2-529acfeee54d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:16:59.795814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 157/305] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:16:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:16:59.800742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:16:59.800742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:16:59.800742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:16:59.814814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:16:59.819229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:16:59.820242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:16:59.843990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 149/306] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 12 02:16:59.845714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.905881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 158/307] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:16:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:16:59.906716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:16:59.976098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0d4c21-d777-4edc-8a33-42a23233b7a9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:16:59.991187 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.021263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6034eb-5446-4c1c-a5e2-b55fd5bde6a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.022229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6034eb-5446-4c1c-a5e2-b55fd5bde6a9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.022457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6034eb-5446-4c1c-a5e2-b55fd5bde6a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.022820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6034eb-5446-4c1c-a5e2-b55fd5bde6a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.032226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a6dc92-0f49-4efc-b796-20ddad501e5b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.078124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43226cf6-cf38-4ea6-8641-42c8de10a069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.079553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43226cf6-cf38-4ea6-8641-42c8de10a069 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.080070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43226cf6-cf38-4ea6-8641-42c8de10a069 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.080475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43226cf6-cf38-4ea6-8641-42c8de10a069 None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.082022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43226cf6-cf38-4ea6-8641-42c8de10a069 None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.082982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.085375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43226cf6-cf38-4ea6-8641-42c8de10a069 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.088060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 159/308] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:16:59 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:00.089522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d16b39e-2942-4969-a503-2f4fb856b98f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.089941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d16b39e-2942-4969-a503-2f4fb856b98f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.090344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d16b39e-2942-4969-a503-2f4fb856b98f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.092260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 160/309] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:00.093552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d16b39e-2942-4969-a503-2f4fb856b98f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.109254 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6034eb-5446-4c1c-a5e2-b55fd5bde6a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.110861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6034eb-5446-4c1c-a5e2-b55fd5bde6a9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.223056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d16b39e-2942-4969-a503-2f4fb856b98f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.256894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 150/310] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:16:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:00.258317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.258561 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.258834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.265363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.266043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.266323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.266590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.267688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.269190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.270938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 161/311] 10.222.0.23 () {56 vars in 1180 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:00.273929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.281769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.282975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:00.284133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:00.370559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 151/312] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:00.370559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.387621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.413493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bdbcc0-a851-4211-85db-80f7776fafd9 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.457642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.458506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.458640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.458994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.472327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:00.477348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:00.478222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:00.478991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 162/313] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:00.479644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.499889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a397b1c-3ee0-43f2-aa61-4bbcff53e383 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.538520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.539181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.539451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.539717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] PATH_INFO: `/v3/projects/239dfc9cbb164dc89983358a6d8450bd` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.540859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] RBAC: Authorizing `identity:get_project(project_id=239dfc9cbb164dc89983358a6d8450bd)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.542315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.544006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 152/314] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/projects/239dfc9cbb164dc89983358a6d8450bd => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:00.546114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.563292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.613081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeb10f6-d87c-4cdf-9127-c0d9274349d8 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.643063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.655052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.655663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.655934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.656200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.657389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] RBAC: Adding query filter params (group.id=a28004e8a06845068d4ab297c642dcb9, role.id=cd7b8279c49f4441b98ba176e7e27199, scope.project.id=239dfc9cbb164dc89983358a6d8450bd) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:00.658009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.659912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3136a8b-78f3-4220-9b4a-c66ee7f88d09 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.667842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 163/315] 10.222.0.23 () {56 vars in 1376 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/role_assignments?group.id=a28004e8a06845068d4ab297c642dcb9&scope.project.id=239dfc9cbb164dc89983358a6d8450bd&role.id=cd7b8279c49f4441b98ba176e7e27199 => generated 611 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:00.668910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc85515-47ca-4297-809a-ff4ec940ca4d None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.669349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc85515-47ca-4297-809a-ff4ec940ca4d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.669642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc85515-47ca-4297-809a-ff4ec940ca4d None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.671514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 164/316] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:00.672229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc85515-47ca-4297-809a-ff4ec940ca4d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.673120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618ee99-df6e-4f02-9aec-abd302175d0c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.728722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.729361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.730149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.730149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.730702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:00.731003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.732530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af27e72-a742-48fa-8762-c9f171b3b7a1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.743337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 153/317] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:00.749795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.750085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.750336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.769662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:00.770911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:00.813861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc85515-47ca-4297-809a-ff4ec940ca4d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:00.854094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:00.861988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 154/318] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:00.862747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.863497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:00.864355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:00.864629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:00.864972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:00.866100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:00.867574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:00.874362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 165/319] 10.222.0.23 () {56 vars in 1180 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:00.875229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:00.956619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.000443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08df8d26-eac2-4f4f-92fa-e873b53e9e4b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.001116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be1cc5e-a7a5-4772-96e2-6ba55c144e51 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.041751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.042409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.042689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.042790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.042976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.043332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.043607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.043876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.057968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:01.064330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:01.065223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:01.066404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 155/320] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:01.067338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.122729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.133119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.134595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.197077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf1f730-5eaf-489b-a96a-5bdfcb95d9aa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.230084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.240954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.241568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.241850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.242118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.242849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] RBAC: Adding query filter params (group.id=a28004e8a06845068d4ab297c642dcb9, role.id=7914a50fd85645bd870c4f7502de5b02, scope.project.id=239dfc9cbb164dc89983358a6d8450bd) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:01.243152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.244589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.252930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 156/321] 10.222.0.23 () {56 vars in 1376 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/role_assignments?group.id=a28004e8a06845068d4ab297c642dcb9&scope.project.id=239dfc9cbb164dc89983358a6d8450bd&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 611 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:01.257850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.272133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 166/322] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:01.273256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.400518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ecff1-10b0-4214-ac3c-6962311d7856 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.404071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.411865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10648907-49e6-4056-84c7-a013ab83ed8c None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.442735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.443360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.443642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.443915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.444582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:01.444932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.446511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.455656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 157/323] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:01.456493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.459101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.459744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.460023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.460290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.473655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:01.479200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:01.480178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 167/324] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:01.481197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.520383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.575449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13021b0-cbe7-457a-b2ee-4de699fb0d7c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.609175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e0c37d-c9e8-4134-bdb6-6437448d4468 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.616262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.617044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.617327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.617720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.618860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.620451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.622168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 158/325] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:01.623016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.651241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.651241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.651241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.651241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.683866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.740227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.740227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.740979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9258031c-9517-4260-92e0-79f43f1bde6d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.767149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.780249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.780920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.781198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.781473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.782132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:01.782436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.783821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.793843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 159/326] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:01.795553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.861603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 168/327] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:01.865610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.896461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c27658-4f5d-41c5-bf59-5c94b436ec50 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.923236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:01.929763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:01.930387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:01.930661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:01.930928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:01.932103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:01.933612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:01.935179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 160/328] 10.222.0.23 () {56 vars in 1180 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:01.936016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:01.968837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275eb354-70dc-4eff-bc8a-989281ec22eb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:01.989284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:02.008707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.009346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.009619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.009885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.048833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7ecbfd-d528-4b22-87b0-de9eb18a6cf0 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.075661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.077190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.081495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.082114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.082382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.082642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] PATH_INFO: `/v3/projects/476619be12f54d90aed177e72373e0d9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.083734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] RBAC: Authorizing `identity:get_project(project_id=476619be12f54d90aed177e72373e0d9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.085172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.086772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 161/329] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/projects/476619be12f54d90aed177e72373e0d9 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:02.087518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:02.141615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:02.183201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 169/330] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:02.184237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:02.187414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147cc4fe-a387-4821-8915-3bf52c2c9955 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.206517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:02.221464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.222068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.222345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.222613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.223334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] RBAC: Adding query filter params (group.id=a28004e8a06845068d4ab297c642dcb9, role.id=7914a50fd85645bd870c4f7502de5b02, scope.project.id=476619be12f54d90aed177e72373e0d9) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:02.223640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.225093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.230700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 162/331] 10.222.0.23 () {56 vars in 1376 bytes} [Wed Oct 12 02:17:02 2022] GET /identity/v3/role_assignments?group.id=a28004e8a06845068d4ab297c642dcb9&scope.project.id=476619be12f54d90aed177e72373e0d9&role.id=7914a50fd85645bd870c4f7502de5b02 => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:02.231525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:02.300772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1757501-aaf7-4774-852f-61f7e659d1ea None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.338722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.339368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.339651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.339903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] PATH_INFO: `/v3/groups/a28004e8a06845068d4ab297c642dcb9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.341085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] RBAC: Authorizing `identity:get_group(group_id=a28004e8a06845068d4ab297c642dcb9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.342678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.349101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 170/332] 10.222.0.23 () {56 vars in 1180 bytes} [Wed Oct 12 02:17:02 2022] GET /identity/v3/groups/a28004e8a06845068d4ab297c642dcb9 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:02.350258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:02.356339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc33bb-c140-4089-8f4a-368b6676ecfa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.384627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:02.395062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.395700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.395973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.396241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] PATH_INFO: `/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.397391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] RBAC: Authorizing `identity:get_project(project_id=1b152cd4fded41fa8ba95182a5dc98e4)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.398833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.400635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 163/333] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:17:02 2022] GET /identity/v3/projects/1b152cd4fded41fa8ba95182a5dc98e4 => generated 344 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:02.406058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:02.471002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadb4ec-04ca-4d3d-a4f4-51867cb2cde6 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.472970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:02.513803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.514507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.514758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.514941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.515653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] RBAC: Adding query filter params (group.id=a28004e8a06845068d4ab297c642dcb9, role.id=cd7b8279c49f4441b98ba176e7e27199, scope.project.id=476619be12f54d90aed177e72373e0d9) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:02.515950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.517420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.524513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 171/334] 10.222.0.23 () {56 vars in 1376 bytes} [Wed Oct 12 02:17:02 2022] GET /identity/v3/role_assignments?group.id=a28004e8a06845068d4ab297c642dcb9&scope.project.id=476619be12f54d90aed177e72373e0d9&role.id=cd7b8279c49f4441b98ba176e7e27199 => generated 611 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:02.537397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974820f2-cb3e-4ab6-8fe2-f7f4eff332af None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.576898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc9eeb6-abe4-4cab-a944-e7ff649c7ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.577799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9eeb6-abe4-4cab-a944-e7ff649c7ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.577799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9eeb6-abe4-4cab-a944-e7ff649c7ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.577983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9eeb6-abe4-4cab-a944-e7ff649c7ec8 None admin] PATH_INFO: `/v3/groups/5418044205d548299c107e00d7c5d070` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.580224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc9eeb6-abe4-4cab-a944-e7ff649c7ec8 None admin] RBAC: Authorizing `identity:get_group(group_id=5418044205d548299c107e00d7c5d070)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.581763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc9eeb6-abe4-4cab-a944-e7ff649c7ec8 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.583682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 164/335] 10.222.0.23 () {56 vars in 1180 bytes} [Wed Oct 12 02:17:02 2022] GET /identity/v3/groups/5418044205d548299c107e00d7c5d070 => generated 229 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:02.589746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:02.690010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:02.699927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf548e0-d00f-4192-872b-36c2dc1257c0 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:02.735183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:02.735786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:02.736057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:02.736327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:02.737074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] RBAC: Adding query filter params (group.id=5418044205d548299c107e00d7c5d070, role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=1b152cd4fded41fa8ba95182a5dc98e4) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:02.737380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:02.738828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376eb171-92e2-4587-bacb-3f5c027567aa None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:02.745076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 172/336] 10.222.0.23 () {56 vars in 1376 bytes} [Wed Oct 12 02:17:02 2022] GET /identity/v3/role_assignments?group.id=5418044205d548299c107e00d7c5d070&scope.project.id=1b152cd4fded41fa8ba95182a5dc98e4&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:05.279151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81d7b20-dbcb-4516-ac01-56bd6b6579f7 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.279651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81d7b20-dbcb-4516-ac01-56bd6b6579f7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.279723 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81d7b20-dbcb-4516-ac01-56bd6b6579f7 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:05.281297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 165/337] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:05.286718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528af6d-36cb-4acd-a2d6-080801c00f97 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.287118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528af6d-36cb-4acd-a2d6-080801c00f97 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.287284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528af6d-36cb-4acd-a2d6-080801c00f97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:05.309952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5528af6d-36cb-4acd-a2d6-080801c00f97 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:05.311646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5528af6d-36cb-4acd-a2d6-080801c00f97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:05.400965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 173/338] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:17:05.585128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35d7cd-9f4a-4c86-9ae1-89301b519ce9 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.585788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35d7cd-9f4a-4c86-9ae1-89301b519ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.586287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35d7cd-9f4a-4c86-9ae1-89301b519ce9 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:05.588101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 166/339] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:05.594082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.594764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.595233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:05.625276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:05.626793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:05.693682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a35d7cd-9f4a-4c86-9ae1-89301b519ce9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:05.726328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 174/340] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:05.811357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a35d7cd-9f4a-4c86-9ae1-89301b519ce9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:05.832887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:05.851725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:05.852363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.852637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.852930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:05.867690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:05.869177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-dda9a5d2-3bae-4552-949c-4d10529ee923 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:05.887265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:05.888256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:05.889152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:05.889152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:05.889152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:05.892803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 167/341] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:05.894491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:05.926198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:05.978600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0defb00a-555c-4b83-8faa-700dc752974f None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.978861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0defb00a-555c-4b83-8faa-700dc752974f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.979075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0defb00a-555c-4b83-8faa-700dc752974f None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:05.980379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 168/342] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:05.997012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864ca72-77b7-4825-8c0e-7ab5e563554b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:05.997316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864ca72-77b7-4825-8c0e-7ab5e563554b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:05.998074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864ca72-77b7-4825-8c0e-7ab5e563554b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.025559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8864ca72-77b7-4825-8c0e-7ab5e563554b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:06.026742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8864ca72-77b7-4825-8c0e-7ab5e563554b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:06.066886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56d9976-ef18-4e4f-81a3-f3a22c748c07 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.108908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.109907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.110541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.111301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.112161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:06.112676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.114764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea951ad5-93d4-4f50-b88d-932e9340d7c1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.127890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 169/343] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:17:06.132186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b8c302-6c11-4284-adf5-4a4fedfba361 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.132186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b8c302-6c11-4284-adf5-4a4fedfba361 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.132186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b8c302-6c11-4284-adf5-4a4fedfba361 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.132186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 170/344] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:06.132186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b8c302-6c11-4284-adf5-4a4fedfba361 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.141713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 175/345] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:06.142968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.143680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.143960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.165929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:06.167178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:06.244987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 176/346] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:06.245770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.251152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:06.266582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b8c302-6c11-4284-adf5-4a4fedfba361 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.268662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:06.322897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.324304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.324828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.325315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.346004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.347707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:06.382030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:06.383048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:06.383970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:06.383970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:06.383970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:06.385985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 171/347] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:06.392232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.425284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5662c48b-e553-4fa8-a3ce-e51ee1c24833 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.473733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.474524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.474872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.475216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.475937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.478095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.505418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 177/348] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:17:06 2022] POST /identity/v3/users => generated 246 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:06.506618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.532272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b0016-8f4e-4c68-8896-b96982bb4004 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.574252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.574973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.575259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.575595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.576318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:06.576623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.578205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.582356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:06.594872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 172/349] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:06.595583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.614282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:06.636628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9853c4c5-ebea-483a-a103-020f21af3051 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.677025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.678096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.678532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.679048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.694951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.697472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.716922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a28dd-7b03-4240-a24a-254f63b8efad None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:06.719054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:06.719971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:06.720888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:06.720888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:06.720888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:06.721407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 178/350] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:06.722027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.761287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.762175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.762445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.762716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.773103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.774546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:06.793093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:06.794030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:06.795388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:06.795388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:06.795388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:06.799870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 173/351] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:06.800790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.863497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1db6d18-e66d-4fe2-8c08-49bee3cda90d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.875094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:06.879343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:06.905596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.906891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.907418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.907959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.908880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.910933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.913318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdd509-f858-437a-92a0-e594225cf570 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:06.945546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 179/352] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:17:06 2022] POST /identity/v3/users => generated 248 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:06.947294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:06.972914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:06.974680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:06.974956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:06.975225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:06.978867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:06.980260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:06.982380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:06.999666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 174/353] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:07.005471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:07.146509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed266f1-c2fe-45ca-bd7c-793eaabe6623 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:07.148954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c663ab-8919-4a93-b8d2-2dfa84f4d61f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:07.184532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:07.193508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:07.194129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:07.194408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:07.194677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:07.195324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:07.195623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:07.196635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ed96b9-98e3-4e65-ba8f-3b557628007e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:07.197035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3bd978-7014-455e-9f23-1b572682eea9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:07.197284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed96b9-98e3-4e65-ba8f-3b557628007e None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:07.197559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed96b9-98e3-4e65-ba8f-3b557628007e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:07.197827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed96b9-98e3-4e65-ba8f-3b557628007e None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:07.198468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ed96b9-98e3-4e65-ba8f-3b557628007e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:07.199876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ed96b9-98e3-4e65-ba8f-3b557628007e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:07.211740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 180/354] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:07.213432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144cb70-09f2-4dc5-9780-f8e913855824 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:07.213764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144cb70-09f2-4dc5-9780-f8e913855824 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:07.213986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144cb70-09f2-4dc5-9780-f8e913855824 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:07.215327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 181/355] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:07.220172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144cb70-09f2-4dc5-9780-f8e913855824 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:07.245038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 175/356] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:17:07 2022] POST /identity/v3/users => generated 249 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:17:07.248286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:07.250649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:07.253082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:07.270843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:07.298957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:07.300267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:07.427461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 176/357] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:07.442066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144cb70-09f2-4dc5-9780-f8e913855824 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:07.479532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:07.480188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:07.480457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:07.480793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:07.481445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:07.482996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:07.507552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 182/358] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:17:07 2022] POST /identity/v3/users => generated 248 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:07.708556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:07.815243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bb7244-c1c0-4950-a194-1f87ab264687 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:07.829979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:07.857587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:07.858211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:07.858487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:07.858752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:07.859424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:07.860811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:07.872958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.sql.core [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 12 02:17:07.872958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 12 02:17:07.872958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [parameters: {'id': 'c650a4a162944656992ae4354a95d689', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 12 02:17:07.872958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81364) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:17:07.879986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return ref.to_dict() Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:17:07.880873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application yield resource Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.rollback() Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.force_reraise() Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise self.value Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application session.commit() Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 12 02:17:07.881784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._prepare_impl() Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.flush() Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._flush(objects) Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application compat.raise_( Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application flush_context.execute() Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rec.execute(self) Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 12 02:17:07.882623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:17:07.883482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:17:07.884272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [parameters: {'id': 'c650a4a162944656992ae4354a95d689', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:07.885367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 12 02:17:07.887052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:07.888985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 177/359] 10.222.0.23 () {60 vars in 1131 bytes} [Wed Oct 12 02:17:07 2022] POST /identity/v3/roles => generated 144 bytes in 173 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:07.897896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:08.014503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:08.033687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd603f-e361-49cc-bab7-09395dfcad5f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:08.076095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9fcff6-cb87-4062-a97d-4fdaaf0835e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:08.077096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fcff6-cb87-4062-a97d-4fdaaf0835e4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:08.077603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fcff6-cb87-4062-a97d-4fdaaf0835e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:08.078095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fcff6-cb87-4062-a97d-4fdaaf0835e4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:08.095153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9fcff6-cb87-4062-a97d-4fdaaf0835e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-3b9fcff6-cb87-4062-a97d-4fdaaf0835e4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:08.106111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:08.107095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 12 02:17:08.107887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:08.116227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 183/360] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:08.118840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:08.256486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6592544-6da8-4909-baa1-df4849ff142a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:08.303295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e43f81-650a-48af-a282-b5808429aada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:08.303821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e43f81-650a-48af-a282-b5808429aada None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:08.304093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e43f81-650a-48af-a282-b5808429aada None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:08.304355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e43f81-650a-48af-a282-b5808429aada None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:08.305035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e43f81-650a-48af-a282-b5808429aada None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:08.305334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e43f81-650a-48af-a282-b5808429aada None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:08.306701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e43f81-650a-48af-a282-b5808429aada None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:08.320671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 178/361] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:08.980090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cebcd2-7099-45fc-8687-2499ab54dbf0 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:08.981123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cebcd2-7099-45fc-8687-2499ab54dbf0 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:08.981621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cebcd2-7099-45fc-8687-2499ab54dbf0 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:08.983514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 184/362] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:08.988524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd910a43-ab6f-49a6-81f9-941c82ffa9f7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:08.989382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd910a43-ab6f-49a6-81f9-941c82ffa9f7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:08.989872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd910a43-ab6f-49a6-81f9-941c82ffa9f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:09.010213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cd910a43-ab6f-49a6-81f9-941c82ffa9f7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:09.011740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cd910a43-ab6f-49a6-81f9-941c82ffa9f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:09.046929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:09.091076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 179/363] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:09.392622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9c526-120b-4ec0-9733-c06b5678fd02 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:09.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9c526-120b-4ec0-9733-c06b5678fd02 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:09.393422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9c526-120b-4ec0-9733-c06b5678fd02 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:09.394873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 185/364] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:09.400143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:09.400428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:09.400729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:09.429677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:09.430860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:09.519267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 180/365] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:09.576233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9c526-120b-4ec0-9733-c06b5678fd02 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:09.731596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9c526-120b-4ec0-9733-c06b5678fd02 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:09.781386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:09.782420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:09.782920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:09.783408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:09.800400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:09.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:09.807057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:09.807837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:09.810565 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 186/366] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:09.816108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:09.927088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:09.930913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a5e909-7c59-47f6-b316-fa7487c126b0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:09.968907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:09.969726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:09.970002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:09.970270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:09.970955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] RBAC: Adding query filter params (name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:09.971327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:09.972763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:09.982921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 181/367] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:09.989590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:09.990309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.001751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.139886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb9c5d-5fe8-41dc-bfa1-d95fbc85d548 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.147819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f864b7-43d8-42cb-a5e6-662a8be385f8 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.183327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dae942e-6c8e-4f6e-b85d-0cfa48a15607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.183533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.183970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dae942e-6c8e-4f6e-b85d-0cfa48a15607 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.184165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.184262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dae942e-6c8e-4f6e-b85d-0cfa48a15607 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.184438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.184538 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dae942e-6c8e-4f6e-b85d-0cfa48a15607 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.184722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.195220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dae942e-6c8e-4f6e-b85d-0cfa48a15607 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:10.195416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:10.196864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-1dae942e-6c8e-4f6e-b85d-0cfa48a15607 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:10.200513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:10.201411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:10.202217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 182/368] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:10.203403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cba6e6-b589-4a20-9f29-c9abea723a02 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.203403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cba6e6-b589-4a20-9f29-c9abea723a02 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.203403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cba6e6-b589-4a20-9f29-c9abea723a02 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.204789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 183/369] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:10.205600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a726ac-609a-4a37-8e1f-3363fe1fb757 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.205900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a726ac-609a-4a37-8e1f-3363fe1fb757 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.206177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a726ac-609a-4a37-8e1f-3363fe1fb757 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.207516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 184/370] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:10.211918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a726ac-609a-4a37-8e1f-3363fe1fb757 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-8cbb65c6-df76-4d41-a37d-866f80fca189 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:10.221204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:10.222107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:10.222940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:10.222940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:10.222940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:10.222940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 187/371] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:10.226746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c603987-9bd6-4417-9970-069beb0e1640 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.226746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c603987-9bd6-4417-9970-069beb0e1640 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.226746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c603987-9bd6-4417-9970-069beb0e1640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.234577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.244685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6c603987-9bd6-4417-9970-069beb0e1640 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:10.245906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6c603987-9bd6-4417-9970-069beb0e1640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:10.298207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.315727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 188/372] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:10.316829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.317114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.317380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.320363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a726ac-609a-4a37-8e1f-3363fe1fb757 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.334832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:10.336008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:10.363730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.364513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.364637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.364906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.365573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] RBAC: Adding query filter params (name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:10.365953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:10.367238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:10.384277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 185/373] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:10.385207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.416664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 189/374] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:10.417908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.485863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.523408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce77ade-0970-425b-b107-d9b7746a98aa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.543709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506aeaf3-5f84-414a-b718-65308ba75c55 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.563137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.584979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.586052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.586332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.586886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.596372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.597035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.597310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.597577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.598209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:10.598501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:10.599846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:10.600184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:10.602701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:10.611978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 186/375] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:10.612680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:10.624230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:10.625158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:10.626108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:10.626108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:10.626108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:10.628214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 190/376] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:10.633515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.743981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208237da-3071-4951-adcd-28d4562514f1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.750545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931cf1e2-fe1a-4e74-9213-55f501919357 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.780536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.781234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.781535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.781782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] PATH_INFO: `/v3/users/nova` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.789413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:10.790881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:10.791959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.792796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.793204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.793606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:10.803499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:10.804445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:10.805717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:10.806224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:10.808844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 191/377] 10.222.0.23 () {56 vars in 1093 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:10.809553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:10.810623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:10.811625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 187/378] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:10.812920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.813616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.816552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.847000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:10.920669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f6694f-3dc6-4fca-b72f-24216d9014a7 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.922280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d21e61-17a2-4948-a45c-bfb52cb4fffb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:10.956309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.957031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.957265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.957531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.958161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:10.958752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:10.959029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:10.959301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:10.959934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:10.960244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:10.961668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:10.968480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:10.973778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:10.974936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:10.976328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:10.977139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 192/379] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:10.979691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 188/380] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:10.979856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:10.980391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.048126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.072440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.084891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36d95c-ddd2-495d-b897-2fe4b01a40b6 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.093303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c554f0-90f3-4df9-a900-bd5fce279657 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.126091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.126970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.127245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.127506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.128220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:11.128515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.130035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.135596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.136205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.136477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.136774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.137438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] RBAC: Adding query filter params (name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:11.137733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.139136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.147278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 189/381] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:11.148057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.176369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 193/382] 10.222.0.23 () {56 vars in 1133 bytes} [Wed Oct 12 02:17:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:11.176369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.262608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda211d-ed97-4f9b-a135-a499b963ae02 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.283543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e851cf-b364-4fc8-bc75-b110a065cbe0 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.288668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.301199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.301971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.302076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.302279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] PATH_INFO: `/v3/users/glance` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.310012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.311454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.322516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.323133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.323411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.323682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.324340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:11.324550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:11.326070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 12 02:17:11.326930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:11.328007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.328077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 190/383] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/users/glance => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:11.328077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.331181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.339724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 194/384] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:11.340526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.353799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.432635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cc8e6a-efd4-4478-b6b2-7ae07d779f7f None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.445142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300b5b2-d3d8-4021-8a50-5c31f415170f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.465284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.465963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.467481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.467481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.477765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.478658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.479219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.479327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.479575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.479838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.490116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.491555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:11.493601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:11.494474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:11.495470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:11.495470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:11.495470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:11.495470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 191/385] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:11.495470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:11.508445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:11.509536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:11.510455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:11.510455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:11.510455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:11.510455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 195/386] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:11.510455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.532617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.572205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.597177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e784afd3-3e09-4448-9e19-54d24bc54a65 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.613737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10161790-e95c-418e-8074-de026c6464b6 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.630503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.631097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.631368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.631636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.632342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:11.632638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.634065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2517a0-8b15-49ff-b5ee-5481ef1bb067 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.646192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.646802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.647076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.647341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.658047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.659490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.669964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 192/387] 10.222.0.23 () {56 vars in 1137 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:11.671245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ac7abb-10d1-4e95-acec-113151e2f236 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.671528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ac7abb-10d1-4e95-acec-113151e2f236 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.671793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ac7abb-10d1-4e95-acec-113151e2f236 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.674742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 193/388] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:11.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:11.675690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:11.676524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:11.676524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:11.676524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:11.676682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ac7abb-10d1-4e95-acec-113151e2f236 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.679112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 196/389] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:11.679959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.766281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.774211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ac7abb-10d1-4e95-acec-113151e2f236 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.778344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95955d25-2e38-4dc5-b4b0-0ae20cd2228d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.791171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:11.808208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.808843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.809124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.809396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.810542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:11.810841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.812224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.813604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.815593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.816818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.817176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.817832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:11.818138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.819548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77786f-6a80-486f-bf9c-679ccbbe5aa7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:11.823478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 194/390] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:11.824211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.833736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 197/391] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:11.834959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.835240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.835502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.853745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:11.854931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:11.920165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 198/392] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:11.921149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:11.937591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb443784-0a57-45b1-9a4a-93881689ce06 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:11.975969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:11.977037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:11.977315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:11.977584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:11.988740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:11.990389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:12.005590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.006498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:12.007331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:12.007331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:12.007331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:12.007331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 195/393] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.007546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.016989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.033348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09ea453-2c82-4ceb-bdd6-8f305a88663b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.034903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.067913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.068511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.068811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.069077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.069731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:12.070032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.071415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.088200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 199/394] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:12.088200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.124722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5697d008-aaa4-4b34-8616-1844ead451d2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.157214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.157836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.158106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.158369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.165748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.167498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:12.179589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.180683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 196/395] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/users/neutron => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.181782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.209836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f858e9f1-a588-4e5f-8b21-1c6fae238d64 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.242691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.243316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.243572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.243691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.243850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.251418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.252899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:12.264457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.265513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:12.266369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 200/396] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.266835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.280562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4637bee-910b-4168-9d72-c71395a6ee99 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.291839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.315972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.316572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.316885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.317149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.317777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:12.318074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.319429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.331173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 197/397] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:12.331952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.369620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b6e31-cc3d-4cc1-be71-76db21ffe064 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.401853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.402464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.402734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.403000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.410664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.412095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:12.423422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.424386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 201/398] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/users/cinder => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.425324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.429845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036bef6-85f8-46bb-b596-8c14f16f0741 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.455368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.464073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.464804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.465080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.465347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.466580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:12.466877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.468295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.485358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 198/399] 10.222.0.23 () {56 vars in 1139 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:12.486157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.510975 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.529622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a205248-e3f8-4762-a70d-02a2f5c8e8d2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.562526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.563149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.563424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.563694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.574766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.576806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.584868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40229b84-ad05-4f8d-8eac-d6b074315a42 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:12.590775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.591871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:12.592928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:12.592928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:12.592928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:12.596823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 202/400] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.597631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.619369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.620027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.620242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.620508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.621210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:12.621506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.623482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.640602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 199/401] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:12.641415 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.688295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.695857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f83de5-8194-4333-91a0-0c077839eae4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.729662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.730692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.730968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.731236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.733783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.739481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.741431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.742335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44468656-e860-4595-86dd-3a13c65f0961 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:12.752867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.753846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 203/402] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.754706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.780222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.780868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.781140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.781402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.782121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:12.782421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.783837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.800238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 200/403] 10.222.0.23 () {56 vars in 1137 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:12.801053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.857470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e85164-de4e-416c-80c5-686800ca2db9 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.890190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.890738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.891010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.891284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.896761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7cd2a3-793e-4d84-ac48-2c6796da8ae0 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:12.898259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:12.902397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.904424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:12.918355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:12.919360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:12.920195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:12.920195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:12.920195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:12.920195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 204/404] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:12.920195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.935018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:12.935649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:12.935925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:12.936191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:12.936850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:12.937148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:12.938495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:12.949678 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 201/405] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:12.953551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:12.965659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.019094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e75d3fc-7904-4e7a-9ed8-1ab905eee5a0 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.051118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c7dcc-729c-4d46-8747-72f61194b118 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.051356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.051997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.052268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.052533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/aa015e7713c34f16be4529081c849ccf/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.066219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=aa015e7713c34f16be4529081c849ccf, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.067658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.081760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 205/406] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:12 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/aa015e7713c34f16be4529081c849ccf/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:17:13.082616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.084581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.085536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.086732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.087025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.087740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:13.088090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.089603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.105874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 202/407] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:13.107401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.136552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.204990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.211858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c796256-f541-4dbd-80ad-774bea86b759 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.226682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12ccc3d-084f-4a24-adec-b8f8fb275ae4 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.255242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.255928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.256208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.256476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.267348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.268219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.268851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.269058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.269432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.269924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.270664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:13.271100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.272586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:13.284800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:13.285990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:13.290226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:13.290226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:13.290226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:13.290617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 203/409] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:13.290706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 206/409] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:13.290772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.290845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.376312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.395701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99d64f-c340-4774-ba82-0df6ed66432b None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.398154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd60b9-61bf-4e02-9eae-2c0dcc0edf4b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.431908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.432561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.432867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.433134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/3d2788633742480981376ed8b171ac89/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.435274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.435914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.436188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.436456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.437137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.438596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.448754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=3d2788633742480981376ed8b171ac89, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.450188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.458270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.460875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 204/410] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:17:13 2022] POST /identity/v3/users => generated 247 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:13.461536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.464895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 207/411] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:13 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/3d2788633742480981376ed8b171ac89/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:17:13.466571 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.609655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55387da-6eed-4b3c-a9fb-b3c7eab50110 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.679413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.680115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.680394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.680738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.689313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.690600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.706174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51571fa0-84ca-41b4-ba9a-853f25e64dd5 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:13.708136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:13.709155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:13.710017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 208/412] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:13.715078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.748217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.749226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.749604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.749878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.750520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:13.750830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.752875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.760900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.768943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 205/413] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:13.774822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.792155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:13.847431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310ed014-c23a-4e3e-8554-40de2772a341 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.891505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66921164-882f-430b-a48c-8d428ffd7030 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:13.893426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.894108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.895340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.895732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.896503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:13.896897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.898406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.922591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 209/414] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:13.928994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:13.950578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:13.951578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:13.952044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:13.952512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:13.962994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:13.965514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:13.978293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:13.979143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:13.979980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:13.982126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 206/415] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:13.987826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:14.097858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:14.147184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637b7d06-6b89-439c-b3bb-a364eec174cb None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:14.179403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd01645f-85ce-485e-b56d-4d26ed8553c5 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:14.196178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:14.197071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:14.197350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:14.197618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:14.198315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:14.198612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:14.199867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:14.200064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:14.228544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b0b113-d985-4636-a9a4-594f89105c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:14.229915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0b113-d985-4636-a9a4-594f89105c2b None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:14.230384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0b113-d985-4636-a9a4-594f89105c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:14.230590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 207/416] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:14.231073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0b113-d985-4636-a9a4-594f89105c2b None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/8c5cd3df4be042f7ac338b9a6b0a50da/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:14.233487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:14.262344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b0b113-d985-4636-a9a4-594f89105c2b None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=8c5cd3df4be042f7ac338b9a6b0a50da, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:14.264954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b0b113-d985-4636-a9a4-594f89105c2b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:14.286828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 210/417] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:13 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/8c5cd3df4be042f7ac338b9a6b0a50da/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:17:14.378142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331d3802-4edf-4126-a8f6-34c68aa3bcd2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:14.404213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:14.427952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9527572-cc9d-4354-a827-2e0b9dacc3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:14.428810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9527572-cc9d-4354-a827-2e0b9dacc3a6 None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:14.429181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9527572-cc9d-4354-a827-2e0b9dacc3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:14.429594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9527572-cc9d-4354-a827-2e0b9dacc3a6 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/575abe426dae4b868bb1c35537a9be35/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:14.447374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9527572-cc9d-4354-a827-2e0b9dacc3a6 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=575abe426dae4b868bb1c35537a9be35, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:14.449013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9527572-cc9d-4354-a827-2e0b9dacc3a6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:14.464178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 208/418] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:14 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/575abe426dae4b868bb1c35537a9be35/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:17:15.607611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870fe2a-527a-4abf-b629-0cf4bbf1c1a8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:15.608149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870fe2a-527a-4abf-b629-0cf4bbf1c1a8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:15.608149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870fe2a-527a-4abf-b629-0cf4bbf1c1a8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:15.609882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 211/419] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:15.615123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb9f194-6c7f-4966-93bf-d651fc5f0a31 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:15.615634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb9f194-6c7f-4966-93bf-d651fc5f0a31 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:15.615634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb9f194-6c7f-4966-93bf-d651fc5f0a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:15.638517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-fbb9f194-6c7f-4966-93bf-d651fc5f0a31 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:15.639763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-fbb9f194-6c7f-4966-93bf-d651fc5f0a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:15.732172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 209/420] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:16.105793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174fd907-bac0-4841-a5a4-e77394e95fa4 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.106438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174fd907-bac0-4841-a5a4-e77394e95fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.107476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174fd907-bac0-4841-a5a4-e77394e95fa4 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.109291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 212/421] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:16.114487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.114771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.115036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.141673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:16.142866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:16.145094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d0741-2963-4937-9b82-60140e8e215b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.145373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d0741-2963-4937-9b82-60140e8e215b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.145645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d0741-2963-4937-9b82-60140e8e215b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.147098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 213/422] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:16.152027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.152302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.152574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.175330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:16.177117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:16.230976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 210/423] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:16.232528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.252758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 214/424] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:17:16.266655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:16.340583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee3485d-da99-4db7-b184-46a1f1576bb4 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:16.375742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309542b3-463b-407b-8e97-52897641049d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:16.376663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309542b3-463b-407b-8e97-52897641049d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.377188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309542b3-463b-407b-8e97-52897641049d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.377672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309542b3-463b-407b-8e97-52897641049d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.388028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309542b3-463b-407b-8e97-52897641049d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-309542b3-463b-407b-8e97-52897641049d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:16.393042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:16.393912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:16.394689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:16.396878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 211/425] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:16.402814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.530939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:16.535361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8d02dd-954d-4a3f-9db9-8a9561be408e None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:16.573387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309542b3-463b-407b-8e97-52897641049d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.607523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:16.608141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.608668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.608668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.609373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:16.609675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:16.611329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:16.619553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 215/426] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:16.620356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.686068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:16.701459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309542b3-463b-407b-8e97-52897641049d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:16.733576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2807aabc-48d0-4c49-968e-c43f0ecfed11 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:16.743181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:16.744417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.745080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.745661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.757959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:16.764455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:16.765349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:16.766152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:16.768738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:16.769441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.769723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.770032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.772785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 212/427] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:16.773832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.783616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:16.789163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:16.790036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:16.790856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 216/428] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:16.791783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.820573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:16.893258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f844f9-218b-4a94-82b7-6c1163b295ba None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:16.908954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5cf6fe-4a5f-4b71-83c8-268cdb676277 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:16.931050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:16.932111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.932390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.932661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.946233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:16.946953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:16.947694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:16.947810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:16.948139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:16.948513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:16.949338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:16.950001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:16.951614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:16.957339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:16.961812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 217/429] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:16.962907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:16.971098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:16.973269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:16.975346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:16.975346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:16.975346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:16.975346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 213/430] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:16.977348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.061612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.082086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b54e4-b794-4c83-85d2-9776f1f562bd None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.123121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dbfd5d-0c6b-4aef-aa40-66306b10002b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.130638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.131827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.132108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.132380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.133080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:17.133777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.135536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.148588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 218/431] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:17.148588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.164213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.165082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.165497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.165905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.177783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.179262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-30cef373-80b4-4ed1-99d8-29d91df918db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:17.195744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:17.196691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:17.197882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:17.197882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:17.197882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:17.197882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 214/432] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:17.198101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833747cf-fe12-451a-80ec-6e7e6dd4f758 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.198347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833747cf-fe12-451a-80ec-6e7e6dd4f758 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.198894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833747cf-fe12-451a-80ec-6e7e6dd4f758 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.204464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 215/433] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:17.204464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833747cf-fe12-451a-80ec-6e7e6dd4f758 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.257941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79edf5c-6855-496b-8042-d88474f78338 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.259039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.291234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.291834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.292108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.292376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.293042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:17.293341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.295261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.306556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833747cf-fe12-451a-80ec-6e7e6dd4f758 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.307681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 219/434] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:17.308590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.311908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.345729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.346355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.346633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.346904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.358112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.359558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-fe6189bc-f556-443b-aeca-9816736ac7f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:17.373355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:17.374688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:17.375721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:17.375721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:17.375721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:17.375721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 216/435] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:17.376008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.376092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.376304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.394427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:17.395611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:17.408911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728ecdf-9867-48fb-ad9e-4b4084c73747 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.442264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.442865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.443138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.443406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.444037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:17.444332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.445744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9fb91e-e2fb-49e3-af5d-0bea61a5041e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.457031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 220/436] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:17.457987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6209f192-96cd-4f04-b4ec-b7bc3926699f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.458268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6209f192-96cd-4f04-b4ec-b7bc3926699f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.458538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6209f192-96cd-4f04-b4ec-b7bc3926699f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.459731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 221/437] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:17.460418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6209f192-96cd-4f04-b4ec-b7bc3926699f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.468283 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 217/438] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:17.469272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.506442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.524102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.569140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6209f192-96cd-4f04-b4ec-b7bc3926699f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.575248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93f1329-5431-4e15-ba7b-f00794ec3a43 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.603294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.603911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.604186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.604449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.612675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.612838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.613356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.613573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.613844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.614204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.614477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:17.614776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.616139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-88dec2bd-8b40-427e-9338-a8b0ac5d789a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:17.625869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:17.626768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:17.627644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:17.635035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 218/439] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:17.635035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.635221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 222/440] 10.222.0.23 () {56 vars in 1093 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:17.635221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.635221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.635221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.651315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:17.652590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:17.729087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 223/441] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:17.729087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.737555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.748051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093c963a-fe4d-489c-8e47-994a64cdd062 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.784495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.785145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.785414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.785675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.789306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:17.793121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.794600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:17.805551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:17.806513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 219/442] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:17.807384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.843637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c29e54-d3e4-490d-9bb5-4edcbd1f4666 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.879262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.879958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.880234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.880502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.881247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:17.881548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.882984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.904887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 224/443] 10.222.0.23 () {56 vars in 1133 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:17.904887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.922095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef31b441-cfe2-41d5-a25b-9dbecf3ce5d1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:17.956800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:17.957420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:17.957695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:17.958105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:17.966515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:17.967936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:17.979302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:17.980366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 220/444] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/users/swift => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:17.981439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:17.997934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.015039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101bbb16-8563-4e21-b167-b25a79203bc9 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.028437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.050089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.050711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.050985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.051247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.061757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:18.066521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:18.067573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 225/445] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:18.069678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.084259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e437b-48ec-4635-ad6e-72cbb004db39 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.120159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.120794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.121063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.121327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.122061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:18.122390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.124097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.141205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 221/446] 10.222.0.23 () {56 vars in 1137 bytes} [Wed Oct 12 02:17:17 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:18.141989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.168259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e01a9-10ae-4c16-8297-9e587e87eba2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.201891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.202491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.202763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.203030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.209843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.238954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f56255-7bab-41f9-930a-a23afe3dd706 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.243709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.270598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.272038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.272471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.273103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.273381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.273649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.274356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:18.274658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.276070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.293245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 222/447] 10.222.0.23 () {56 vars in 1135 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:18.294536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.378387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 226/448] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:18.382911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.390909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08963355-1d55-43d6-a51e-1b02a180273e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.419007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.425252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.425875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.426148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.426413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.436707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:18.441468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:18.442359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:18.443152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 223/449] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:18.443825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.453563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.482256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f227b8b-599e-48ee-9d98-5041570bf013 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.517101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.517638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.517914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.518202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.518851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:18.519148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.520620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.528649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 227/450] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:18.529438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.545900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34ea1d4-1853-4946-924b-302e61e9f100 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.579282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.579880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.580159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.580425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.625712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d31dbb-b0d0-45e5-b732-110cf027adcc None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.636484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.646690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.648239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.658782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.659406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.659680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.659948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.661824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.671646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.673169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:18.687170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:18.688048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:18.689055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:18.689055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:18.689055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:18.689055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 228/451] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:18.689055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.746787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 224/452] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:18.748175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.791508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1639afe4-3f77-4df3-b57d-04d6c6d148ba None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.824003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.824620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.824934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.825727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.833854 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.836590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.838090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.843825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8f3d09-508e-46f5-884c-b65642939555 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:18.852093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:18.853033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:18.854082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:18.854082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:18.854082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:18.854082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 229/453] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:18.854082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.880530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.881166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.881438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.881709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.882351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:18.882647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:18.884001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:18.892184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 225/454] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:18.892988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:18.893196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:18.955355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe520b5c-c962-4d65-ad16-6a7c978c62f6 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:18.990042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:18.990650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:18.990919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:18.991185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:18.994052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c8ff4-ddae-4e28-b190-e4e2c3485ef6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.002176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.003639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:19.017235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.018197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:19.019060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:19.019060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:19.019060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:19.019060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 230/455] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.019060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.031197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.031799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.032067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.032338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.032982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.033275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.034630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.046531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 226/456] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.047250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.064414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.110456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.123102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab788fea-8134-4919-a666-0d621003be09 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.148140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e73383-0c05-4125-beff-206a30a7e92f None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.156330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.156984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.157274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.157802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.167630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.169151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.181084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.181680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.181952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.182217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:19.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.183862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:19.185476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:19.185476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:19.185476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:19.185672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.185672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.185672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.189075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 231/457] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.189809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.196053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 227/458] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.196874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.264445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.290810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84885d90-f67d-4e99-a2b4-5ea25874bcf3 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.294367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a02015-57bb-4355-b6a0-8b8aa97e2faf None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.324053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.324616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.324918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.325236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.327066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.327667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.327941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.328212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.328863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.329162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.330520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.335591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.337071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.338892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.342667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 228/459] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.343473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:19.355411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.356303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:19.357763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:19.357763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:19.357763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:19.357763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 232/460] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.357982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.442467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0672a041-6879-42e6-ad32-e53751f033aa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.458096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af03f3a-859f-4b48-81d4-5bf32fbc1a0e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.470993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.476166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.476817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.477091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.477357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.484445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.485909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.491669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.492266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.492545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.492855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.493491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.493794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.495165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:19.496744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.497613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:19.498716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 229/461] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.499202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.509338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 233/462] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.510148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.566306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.596030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f90c4-e50b-4503-a80a-7df81e085e58 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.610441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf912768-11d1-4e75-a2db-d8df6640c3a6 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.629486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.630112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.630389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.630667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.637444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.638923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.644060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.644673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.644977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.645239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:19.650879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.651681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:19.652507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:19.653269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.653338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 230/463] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.653338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.653725 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:19.668090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.669687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:19.670578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 234/464] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/users/neutron => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.671340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.712546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.760029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb230c4-4b32-4d81-95b6-ed63d671dcf5 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.770007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e049d6b0-cb9d-4a01-8e91-979d3a410abd None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.783240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.792793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.793404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.793680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.794460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.795087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.795377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.796777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.805431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.806028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.806304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.806566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.807240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.807538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.808305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 231/465] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.808984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.809161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.821712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 235/466] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.822814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.911671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bfc3c-5581-4449-9946-a71f2cf72852 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.922872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f97e52b-7ecd-41ff-bd32-c98bd872609f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:19.927389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:19.952881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.953508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.953779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.954043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.959013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:19.959615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:19.959892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:19.960159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:19.960594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.960868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:19.961168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:19.962055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.962591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:19.972969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:19.973790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:19.974624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:19.975553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 236/467] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:19.975553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:19.977086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 232/468] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:19.978148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.008620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.081815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07909c2-75e9-4faa-9727-9fe9897045b2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.085250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62414b5-8bb4-4477-9f1f-18e184e82361 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.115345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.115976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.116252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.116517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.123243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.124104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.124375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.124639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.124925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.125448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:20.125747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.126351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.127216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:20.139154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:20.140074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 233/469] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/users/cinder => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:20.141318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.146459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 237/470] 10.222.0.23 () {56 vars in 1139 bytes} [Wed Oct 12 02:17:19 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:20.147283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.181123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.237395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.247580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a937413-4225-4b73-9fea-e1c23e21cb9c None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.253467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272e45ff-782b-4beb-990c-9922a09fd24c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.283669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.284276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.284550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.284847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.285568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=aa015e7713c34f16be4529081c849ccf) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:20.285875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.287316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.288349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.289041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.289315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.289585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/c57115b9feff40629f5081e32ff4bad3/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.293158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 238/471] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/role_assignments?user.id=aa015e7713c34f16be4529081c849ccf&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:20.297655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.307534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=c57115b9feff40629f5081e32ff4bad3, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.309081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.336740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 234/472] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:20 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/c57115b9feff40629f5081e32ff4bad3/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:17:20.336740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.452159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88917a79-3774-4fcc-b728-e9afba95612f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.478087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.508592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdec9b2-001c-4dc6-867c-5c2e332768db None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.510524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.511069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.511345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.511610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.512313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:20.512675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.514193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.530384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 239/473] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:20.530591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.551299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.561944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.562553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.562825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.563090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.564072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:20.564375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.565857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.585074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 235/474] 10.222.0.23 () {56 vars in 1137 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:20.585933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.650682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e694b2-5122-4329-8a52-ac60056fa3ff None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.689320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.690420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.690901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.691374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.714990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bac74f-1cc0-4acf-b39c-722a75707b66 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.753224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.753837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.754295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.754530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:20.755249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=3d2788633742480981376ed8b171ac89) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:20.755576 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.757140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.765430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 236/475] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/role_assignments?user.id=3d2788633742480981376ed8b171ac89&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:20.766328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.770293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.772806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:20.775015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:20.796411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:20.892749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 240/476] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:20.897864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:20.899631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25ecd67-ab86-4caf-9336-c44b9212d27e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:20.937976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:20.938598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:20.938873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:20.939142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.009748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f390b3d-1898-4a3b-8105-826292bc6b3d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.017739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.019359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.029280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:21.037076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:21.050287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.051166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.051454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.051718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.062491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.064088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:21.085342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:21.086264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:21.087108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:21.087108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:21.087108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:21.087108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 241/477] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:21.092095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.145880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 237/478] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:21.151667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.255640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d1605-9022-4c0e-81b5-9a2e6f1bcb4a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.256731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ffc3aa-b97b-40eb-b8ca-b1389bd7b48a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.259882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:21.301219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.301219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.301639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.301802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.302604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:21.302908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.304677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.305955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.306858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.307410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.307777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.318345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 242/479] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:21.322873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.322873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.324103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:21.342108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:21.343079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:21.344042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:21.344042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:21.344042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:21.344042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 238/480] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:21.355911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.377087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:21.477009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea701e1f-a3ba-48f3-891d-cb8e67835bc2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.524825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.525380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.525648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.525909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.538119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.539570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:21.561370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:21.562256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:21.563075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:21.563932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 243/481] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:21.569160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.576358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2345f3e7-7016-44ae-af1e-d025b03323ee None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.642150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.643258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.644396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.645051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.647140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:21.647917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.650182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.674638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 239/482] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:21.681655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.720923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:21.725916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:21.741952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e6e5d-a31a-42cf-8c04-a5bcb4ef5085 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.781961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.782499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.782776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.783046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:21.783739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:21.784044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:21.785523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:21.801238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 244/483] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:21.807431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:21.922628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2380ba-7503-4471-bbe4-bedcb70268d3 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.958029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5f2c4-81de-4efb-b1be-f505b10c0e5a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:21.995426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:21.996064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:21.996340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:21.996606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.004968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:22.006402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:22.019289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:22.020129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:22.021059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 240/484] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:22.026640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:22.038006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:22.038568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.038846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.039120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.039851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=8c5cd3df4be042f7ac338b9a6b0a50da) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:22.040157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:22.041637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:22.063135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 245/485] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:21 2022] GET /identity/v3/role_assignments?user.id=8c5cd3df4be042f7ac338b9a6b0a50da&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:22.120737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:22.148487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca19e49a-699c-4d72-ab95-5876a883b5f6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:22.189756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:22.190386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.190731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.191006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.191708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:22.192009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:22.193485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300ebc65-ef5e-4ce7-ad63-8de1925086e9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:22.207820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 241/486] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:22.217890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:22.247435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:22.345949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b86b3c5-7919-4c0a-9518-473ae996595c None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:22.383163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:22.383769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.384049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.384315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.385074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=575abe426dae4b868bb1c35537a9be35) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:22.385428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:22.386940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401495f6-c64b-42d4-a1b8-27f5ebf5a74d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:22.393577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 246/487] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:22 2022] GET /identity/v3/role_assignments?user.id=575abe426dae4b868bb1c35537a9be35&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:22.807148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d65b75-38de-4c77-a78b-588e9fd2ccc6 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.807527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d65b75-38de-4c77-a78b-588e9fd2ccc6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.807728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d65b75-38de-4c77-a78b-588e9fd2ccc6 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.809304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 242/488] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:22.818179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.818653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.818728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.838189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:22.839398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:22.897936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae0dfe-ed66-4edc-b647-bf17111a6c8d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.897936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae0dfe-ed66-4edc-b647-bf17111a6c8d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.897936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae0dfe-ed66-4edc-b647-bf17111a6c8d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.899964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 243/489] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:22.919718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:22.920047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:22.920253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:22.934107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 247/490] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:22.940287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:22.941519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:23.052597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 244/491] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:23.264256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:23.272934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:23.402540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:23.410740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b54248-a3e0-4e4e-a36f-f57cb4caf293 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:23.464029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:23.464976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:23.465393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:23.465860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:23.473451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:23.491543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:23.496423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:23.497397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:23.498198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:23.502861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 248/492] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:23.509615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:23.515919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cffa949-ad93-4d0b-af7d-ed8ec1af593e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:23.551761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:23.552732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:23.553209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:23.553675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:23.564449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:23.569735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:23.570588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:23.571883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:23.574734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 245/493] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:23.581045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:23.643665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:23.688692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d41efa0-8d61-4800-9cc6-5107988bc07a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:23.707740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edb64b5-e20c-4f7e-bf0d-6d3b602d985d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:23.725122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:23.746504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:23.747655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:23.747935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:23.748253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:23.749199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:23.749498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:23.750895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:23.771774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:23.772406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:23.772681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:23.772973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:23.773655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:23.773953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:23.775414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:23.776598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 246/494] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:23.777893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:23.794660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 249/495] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:23.803186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:23.941199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce6f8d0-4618-489d-8f6c-2a6ed6aed0c6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:23.952368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fabcbd-0be7-4235-8183-f7ff986b8ed1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:23.986922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:23.991132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:23.992035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:23.992776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:23.993325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:23.996491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:23.998088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:23.998359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:23.998872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.007715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.010224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.011740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.013929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c36b1038-6206-4e69-94d1-0a2b0a987781 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:24.027719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:24.028683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:24.029674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:24.029674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:24.029674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-ed87af0e-9432-46f8-b782-e52518902220 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:24.029839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:24.030655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:24.031528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:24.031528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:24.031528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:24.031528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 250/496] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:24.031998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc56b0-787f-4fb4-83c9-aa187d462136 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.032239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc56b0-787f-4fb4-83c9-aa187d462136 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.032506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc56b0-787f-4fb4-83c9-aa187d462136 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.038081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 247/498] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:24.038205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 251/498] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:24.038205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc56b0-787f-4fb4-83c9-aa187d462136 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.039416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.039685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.039946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.059826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:24.061136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:24.104091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.147036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 248/499] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:17:24.147884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.150312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc56b0-787f-4fb4-83c9-aa187d462136 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.185852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.186827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.187327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.187794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.188645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:24.189166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.190744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.203781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 252/500] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:24.209195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.253701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.305150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0507c8-c599-46a5-bc54-393139c06b21 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.343601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f779c-ee79-4bfb-9e29-26e4f4e0ebf2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.372163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.372813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.373092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.373360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.374257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:24.374555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.375940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.394822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.395456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.395733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.395993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.401045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 249/501] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:24.407634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.412918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.414464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:24.434131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:24.435601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:24.437073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:24.442415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 253/502] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:24.442415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.475877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.566008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1866a61-19a3-4458-add6-5073e217a149 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.566518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fa5aa7-9974-4f41-9b65-e44edd2720d7 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.581205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.609665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.610693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.611017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.611299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.611573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.611842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.612126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.612559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:24.612838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.612937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.614323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.623103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.625642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:24.639323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:24.640286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:24.641327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:24.641844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 254/503] 10.222.0.23 () {56 vars in 1135 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:24.641844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.643508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 250/504] 10.222.0.23 () {56 vars in 1093 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:24.644556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.752762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.758092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f2dd7-5e5d-44ee-a0b9-72efb7d8c965 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.763371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4217ca3b-9168-4e1d-a660-5ff5833b4b52 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.794503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.795530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.796063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.796535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.800350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.801003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.801279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.801544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.811104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.812827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:24.828974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:24.830115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:24.831067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:24.831067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:24.831067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:24.831067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 255/505] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:24.831295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:24.844630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.879408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.880940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.939356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ea7e7-7c17-4e26-aa6e-da4ceb9f2803 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:24.972381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:24.973069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.973318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.973545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:24.974261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:24.974560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:24.976001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d91089-963f-4b89-bd3a-c114a8dc0a13 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:24.980538 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:24.994284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 256/506] 10.222.0.23 () {56 vars in 1133 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:24.994843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 251/507] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 12 02:17:24.995328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10dc5f6-6bab-48b1-b157-49c0abe7e994 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:24.995615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10dc5f6-6bab-48b1-b157-49c0abe7e994 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:24.995883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10dc5f6-6bab-48b1-b157-49c0abe7e994 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.001447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 257/508] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:25.001581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.001676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbabc4c-14b9-4af5-a6c2-48a06f98b568 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.001676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbabc4c-14b9-4af5-a6c2-48a06f98b568 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.001676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbabc4c-14b9-4af5-a6c2-48a06f98b568 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.002574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 258/509] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:25.003310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbabc4c-14b9-4af5-a6c2-48a06f98b568 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.068502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.099541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db04d6-4823-4c7e-ba3c-e55cc858fe26 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.104206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbabc4c-14b9-4af5-a6c2-48a06f98b568 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.134052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.134652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.134921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.135183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.135814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:25.136111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.137530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.140793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.141413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.141686 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.141953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.148429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 252/510] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:25.149233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.157256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.158710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-0e721214-bab1-40de-9f3d-7ee541d8ed2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:25.173197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:25.174127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:25.175003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:25.175003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:25.175003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.175003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 259/511] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:25.175487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99927525-1c83-4434-a855-b664c846f838 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.175759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99927525-1c83-4434-a855-b664c846f838 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.175976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99927525-1c83-4434-a855-b664c846f838 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.193739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-99927525-1c83-4434-a855-b664c846f838 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:25.194918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-99927525-1c83-4434-a855-b664c846f838 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:25.202721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.253187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5440d1-9d98-4ece-b8cc-dcba89eb698d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.259538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 260/512] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:25.260710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.260999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.261268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.284438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:25.285654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:25.286178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.286810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.287083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.287349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.299551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.301581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:25.316836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:25.317798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:25.318672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:25.318672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:25.318672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:25.318672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 253/513] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:25.319563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.340004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.364344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 261/514] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:25.365128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.440140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146ef9f0-7fa5-4f6b-b70f-5feb1df5e4b3 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.442529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.481667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.482503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.482884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.483244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.484004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.485708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.491529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a6b456-96c1-4776-85cc-320f70439297 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.516266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 254/515] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:17:25 2022] POST /identity/v3/users => generated 254 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:25.520847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.554222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.555128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.555411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.555757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.557071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:25.557374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.559142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.582702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 262/516] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:25.582702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.642976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.646799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b3ffc-1408-47e8-93e9-26b42e0a7ce3 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.683468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.684090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.684426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.684727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.685434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:25.685712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.687110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.701470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 255/517] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:25.702280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.723301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f1cab-c93c-4e98-9123-1371c50880e0 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.724622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.761717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.762868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.763145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.763411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] PATH_INFO: `/v3/services/network` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.763986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.765419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:25.776809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:25.777813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.778713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 263/518] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/services/network => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:25.779502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.818300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35648689-9494-4ba1-8840-41bb3f4d6bc1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.854468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.855093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.855364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.855625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.862829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.864898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:25.876735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:25.877699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:25.878609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 256/519] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:25.881539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.886525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b98f7c-294d-4a5d-9703-727b2060c3df None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:25.889182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:25.927549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:25.928266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:25.928643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:25.929035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:25.929715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:25.931247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:25.941014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:25.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:25.943177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:25.944469 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 264/520] 10.222.0.23 () {56 vars in 1129 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:25.945013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:25.990314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:26.023918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43986-8cb0-49f8-bf0e-2d5a5f8a45a5 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.065170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.065726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.066012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.066284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.073645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.074454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9941cb7b-3717-415a-b60e-6afed3a81775 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.075629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:26.091322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:26.092237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:26.093169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:26.094150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 257/521] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:26.095147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.116474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.117127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.117765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.117765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.118263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] RBAC: Adding query filter params (name=network) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.118563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.119949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.126248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 265/522] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:26.127021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.161502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:26.205622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802dcfb-5b89-4390-a9bb-2f6524456ea1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.236207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3556e47a-0673-4431-948b-bf88ebab0d75 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.241923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.242601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.242971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.243695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.244447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.244859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.247556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.255668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:26.267430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 258/523] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:26.268244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.272722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.273375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.273650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.273917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.274539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.274838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.276241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.281134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 266/524] 10.222.0.23 () {56 vars in 1124 bytes} [Wed Oct 12 02:17:26 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:26.282149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.376026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:26.382792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d44a2-63d1-4bd8-99b0-75c8f3bfc5aa None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.385158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914c544-d2c2-4996-8b85-92cc664929cf None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.419597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.420219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.420494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.420788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.421427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] RBAC: Adding query filter params (type=network) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.421726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.422206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.423022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.423180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.423614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.424069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.424993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.425470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.427104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.429779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 267/525] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:26 2022] GET /identity/v3/services?type=network => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:26.430605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.454222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 259/526] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:26.455413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.526602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:26.566179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a2712-ce02-47a4-96fe-cf9f03626816 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.590522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a510c8d-9ff0-48ce-b337-428e14c30d15 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.605447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.606903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.607239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.607506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/aa015e7713c34f16be4529081c849ccf/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.609692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=aa015e7713c34f16be4529081c849ccf, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.611156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.626384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 268/527] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:26 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/aa015e7713c34f16be4529081c849ccf/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:17:26.627188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:26.636464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.637584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.638034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.638473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.639396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.639838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.642461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb316e0-076d-4bca-b371-86bfefa020df None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.651612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 260/528] 10.222.0.23 () {56 vars in 1124 bytes} [Wed Oct 12 02:17:26 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:26.658196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:26.761649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12c2f58-9cc3-4143-9724-156a034d337a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:26.803046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:26.804021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:26.804440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:26.804884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:26.805773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=c57115b9feff40629f5081e32ff4bad3) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:26.806226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:26.807914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f446b9-718f-4887-9ffc-54354536a6c7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:26.816205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 269/529] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:26 2022] GET /identity/v3/role_assignments?user.id=c57115b9feff40629f5081e32ff4bad3&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:27.739955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c54cb6-ea98-4252-85e9-2ea629237376 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:27.740508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c54cb6-ea98-4252-85e9-2ea629237376 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:27.740760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c54cb6-ea98-4252-85e9-2ea629237376 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:27.742429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 261/530] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:27.748614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:27.749068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:27.749406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:27.768245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:27.769502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:27.864685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 270/531] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 12 02:17:28.181211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c54cb6-ea98-4252-85e9-2ea629237376 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:28.308395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c54cb6-ea98-4252-85e9-2ea629237376 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:28.343481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:28.367704 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:28.368644 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:28.369137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:28.369703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:28.389304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:28.394306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:28.395203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:28.396061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:28.402082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 262/532] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:28.413799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:28.445043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:28.584559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9adecdf-2aba-479e-b4d0-9c2c14393564 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:28.630701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:28.631668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:28.632148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:28.632614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:28.633513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:28.634013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:28.635589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:28.647938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 271/533] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:28.653525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:28.783657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47b8f1f-ccf5-4a2a-8158-8077c6fb8c44 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:28.836030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:28.837031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:28.837510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:28.837997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:28.850101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:28.853953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e539fe3f-9bbc-429c-ba12-bcf6cea581f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:28.868195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:28.869195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:28.870059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:28.870059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:28.870059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:28.875842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 263/534] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:28.881119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:28.990349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:29.018220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a119fa9b-5aec-4ec4-b6a8-0e5246459f0e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:29.043967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8268d25e-1813-448b-ae66-6fefbef142d7 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.044273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8268d25e-1813-448b-ae66-6fefbef142d7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.048719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8268d25e-1813-448b-ae66-6fefbef142d7 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.048719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 264/535] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:29.052154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.052433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.056722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.075642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:29.075642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:29.112101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:29.121432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:29.122580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.122855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.123123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.124025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:29.124321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:29.125789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:29.155035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 265/536] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:29.172722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 272/537] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:29.174907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:29.419462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504566bd-d77f-43b6-88ee-2ac937d1c049 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:29.482686 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:29.486562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:29.487219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.487490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.487754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.509684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:29.513255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:29.540368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:29.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:29.542337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:29.546421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 266/538] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:17:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 372 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:29.553553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:29.605128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:29.690862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdfe60-04b7-420a-a2f2-5d9cd5992217 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:29.712605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83aa8b9-80b9-4a8d-9162-825fd7fa5686 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:29.742714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b21e70b-02df-494a-a037-49ccaec6c8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:29.743357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b21e70b-02df-494a-a037-49ccaec6c8f3 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.743632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b21e70b-02df-494a-a037-49ccaec6c8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.743897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b21e70b-02df-494a-a037-49ccaec6c8f3 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.744546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b21e70b-02df-494a-a037-49ccaec6c8f3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:29.745958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b21e70b-02df-494a-a037-49ccaec6c8f3 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:29.755315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:29.757096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:29.757697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.758070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.758311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.759115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:29.759537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:29.761177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b95c4b-65d2-48e7-b365-cfcfcf6c6d75 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:29.769176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 273/539] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:29 2022] POST /identity/v3/services => generated 240 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:29.770752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1ec2f-3191-4c7d-85c7-abed184bafde None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.771032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1ec2f-3191-4c7d-85c7-abed184bafde None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.771302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1ec2f-3191-4c7d-85c7-abed184bafde None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.772527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 274/540] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:29.773688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aea6db-5750-4ea5-bb08-8b0f2151e2a4 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.773968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aea6db-5750-4ea5-bb08-8b0f2151e2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.774237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aea6db-5750-4ea5-bb08-8b0f2151e2a4 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.775714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 275/541] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:29.788098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.788413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.792721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.798772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 267/542] 10.222.0.23 () {56 vars in 1149 bytes} [Wed Oct 12 02:17:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:29.803748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2096e-27ff-4310-bbee-3c1bffd3f5c3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.803748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2096e-27ff-4310-bbee-3c1bffd3f5c3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.803748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2096e-27ff-4310-bbee-3c1bffd3f5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.818159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:29.819402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:29.822923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-20e2096e-27ff-4310-bbee-3c1bffd3f5c3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:29.824128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-20e2096e-27ff-4310-bbee-3c1bffd3f5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:29.901667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 276/543] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:29 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:29.902663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:29.928295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 268/544] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:29 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:29.929686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd45b4a-09d6-470c-826f-2715577de0bf None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.929971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd45b4a-09d6-470c-826f-2715577de0bf None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.930545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd45b4a-09d6-470c-826f-2715577de0bf None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.932635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 269/545] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:29.941961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:29.942325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:29.942752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:29.966204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:29.967966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:29.971175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.039208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243074ea-e231-4d20-9dc7-20554104bd8f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.053289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 270/546] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:29 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 12 02:17:30.076768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.089817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.090524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.090804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.091088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.111143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.114597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:30.128863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:30.129794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:30.130668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:30.130668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:30.130668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:30.130668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 277/547] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:30.135740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.220408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.246646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc12d0a-0f2e-4a52-a25f-53cea01cd9a0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.289973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.290675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.291018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.291354 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.292061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:30.292437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.294242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.302071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.312878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 271/548] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:30.312878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.341037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26cc5fb-83be-46fa-b2f8-4035bbf67bb4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.383626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.384239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.384515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.384813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.385466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.386839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.408731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 278/549] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:30 2022] POST /identity/v3/services => generated 251 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:17:30.408731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.434945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d00cd1-6fd2-4ddd-88b7-0200b1e95ade None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.460146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.495730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.496295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.497123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.497395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.511505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:30.518275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:30.519241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:30.520088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:30.524346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 272/550] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:30.525290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.559995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04329032-bd4c-4da0-9048-730d30593d04 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.593739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.607613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.608485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.608795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.609066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.616027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.618711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:30.632023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:30.632949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:30.633851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 279/551] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:30.634321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.654860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2bf135-8ea3-41fb-890a-60f9b14e9aa2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.697518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.698161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.698440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.698708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.714712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:30.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:30.722982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:30.723794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:30.726454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 273/552] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:30.727466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.742469 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.750966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f420796-bce4-43da-910e-1ca6780ba10d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.787127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.787893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.788370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.788665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.789466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:30.789863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.791487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.800986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 280/553] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:30.801911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.859229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2184cf3b-3ac7-430d-8bb6-8315ce3a8f0d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.877836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:30.905800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.907186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.907903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.908334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.909390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:30.909901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.911902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.928248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e133f75-8dbf-496f-bca7-a6b56589bdf2 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:30.937030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 274/554] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:30.938101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:30.968239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:30.969225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:30.970210 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:30.970889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:30.972236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:30.972912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:30.974633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:30.984421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 281/555] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:30.985601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.009015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:31.042152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9846a-fb46-4a5d-9bce-f9e399793026 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.077605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.078230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.078503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.078769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.089365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.091218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:31.106337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:31.107309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:31.108184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:31.108184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:31.108184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:31.108184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 275/556] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:31.114271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.120170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ada64a7-9c83-4051-9d45-5e03bd5f0454 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.143155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:31.161058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.162218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.162704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.163245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/f8d2a8079f40422c8bdb9b73b391d0c9/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.181042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=f8d2a8079f40422c8bdb9b73b391d0c9, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.183051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.200917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 282/557] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:30 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/f8d2a8079f40422c8bdb9b73b391d0c9/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:17:31.202239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.220899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d500194-49a3-49d9-92cf-5422e12cb9f1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.258534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.259385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.259494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.259726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.271292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.272845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.286615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:31.288557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:31.290259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:31.291165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:31.291165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:31.291165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:31.291165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 276/558] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:31.296389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.334720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b5d9e1-b9f0-40bb-a259-5233f54555e4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.376600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.377733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.378222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.378691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.379623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:31.380125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.381265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:31.383422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.398533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 283/559] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:31.406410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.410722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00738f0c-b408-456c-99ac-c1b777c6efcd None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.507448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.508114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.508438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.508734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.509726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:31.509726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.511129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.526491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 277/560] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:31.532005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.560862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220ec777-8bd7-4e15-b181-ab601c95d35e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.593002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:31.602867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.603776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.604254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.604738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.613394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.615134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:31.627141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:31.628058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:17:31.629458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:31.631333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 284/561] 10.222.0.23 () {56 vars in 1093 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:31.641639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.644338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d4de6d-12d3-4978-a2f8-5e7250c91a54 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.680585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.681255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.681530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.681798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.689482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.690932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.692728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:31.706916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:31.707905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:31.708881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 278/562] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:31.714404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.820688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d889fd-ee7d-4322-b715-b34290e28f37 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.857237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.857847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b577027-9f5c-4777-8489-fce2b96dd79d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:31.858025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.858195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.858401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.859123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:31.859425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.861110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.881405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 279/563] 10.222.0.23 () {56 vars in 1135 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:31.886875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.895075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:31.895824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:31.896106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:31.896373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:31.897127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:31.897428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:31.898996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:31.919103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 285/564] 10.222.0.23 () {56 vars in 1133 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:31.926565 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:31.970317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.022397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.055262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351c6846-4a0e-4a5c-87ed-178199c41da1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.113881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3a6a-8c5c-4773-86d5-c27ec8192612 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.128813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.129478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.129749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.130014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.149900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.151349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:32.176729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:32.177728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:32.178603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:32.178603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:32.178603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:32.178603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 280/565] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:32.185299 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.195918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d472696-5573-4481-b680-38ebd97e2614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.197517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d472696-5573-4481-b680-38ebd97e2614 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.197792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d472696-5573-4481-b680-38ebd97e2614 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.198060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d472696-5573-4481-b680-38ebd97e2614 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.288287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d472696-5573-4481-b680-38ebd97e2614 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.289783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d472696-5573-4481-b680-38ebd97e2614 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.307946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b6023-3cf8-4110-aac1-6a4b2c11a2b2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.326107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.336973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.347511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.348508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.349177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.349861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.350691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:32.351132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.352772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7c094-9186-4db2-b7dd-7e404a80e9aa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.370891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 281/566] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:32.373132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92406af1-5950-493c-81fc-c0aa259dabc4 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.373873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92406af1-5950-493c-81fc-c0aa259dabc4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.374331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92406af1-5950-493c-81fc-c0aa259dabc4 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.376053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 282/567] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:32.376945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92406af1-5950-493c-81fc-c0aa259dabc4 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.394300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 286/568] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:31 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:17:32.395678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.395958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.396226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.414103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:32.415299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:32.484819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 287/569] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:32.485735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.489472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92406af1-5950-493c-81fc-c0aa259dabc4 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.530973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.531618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.531888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.532153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.539507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.541067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:32.554543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:32.555447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:32.556357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:32.560424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 283/570] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:32.561884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.584500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.616141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571043e3-1780-4597-a76e-ba73bf16c5bc None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.621120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.662793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.663595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.663966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.664462 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.674898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.675174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693af147-84db-42c8-8f63-09421996ff31 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.676600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:32.692481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:32.693763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:32.694873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:32.694873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:32.694873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:32.695638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 288/571] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:32.701738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.718629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.719873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.720593 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.721095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.721981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:32.722712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.725101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.740174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 284/572] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:32.746317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.840998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d4bd8c-aeee-4cb5-93b0-dd3515cb9e3f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.873903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.891532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:32.897969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d390d932-5c0b-4605-aaa1-d888bb63c959 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:32.909582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.910240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.910517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.910783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.911426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:32.911723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.913130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.940042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 289/573] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:32.941629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:32.957930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:32.958651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:32.958929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:32.959245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/c57115b9feff40629f5081e32ff4bad3/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:32.961215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=c57115b9feff40629f5081e32ff4bad3, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:32.962711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:32.984362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 285/574] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:32 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/c57115b9feff40629f5081e32ff4bad3/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:17:32.985431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.133735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504a0d8-91ac-4d58-8f16-4f582e454237 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.165186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b9af64-2a36-4c4e-acdd-d621535acc39 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.181164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.182679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.182679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.182679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] PATH_INFO: `/v3/services/network` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.183556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.185490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-08f0c6a5-749c-462a-9ecc-a5260c678881 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:33.197434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:33.198420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:33.199254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 290/575] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/services/network => generated 88 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:33.200574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e2180-a41d-4697-91d0-b63bc3793c72 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.200574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e2180-a41d-4697-91d0-b63bc3793c72 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.204836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e2180-a41d-4697-91d0-b63bc3793c72 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.206742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 291/576] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:33 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:33.207899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e2180-a41d-4697-91d0-b63bc3793c72 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.211876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.212523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.212872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.213135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.221681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.223875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-6869141a-acc5-4d56-905d-46ac5d3a5b9a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:33.237595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:33.238516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:33.239478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:33.243514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 286/577] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:33.245160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.245701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.245985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.265795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:33.267205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:33.269868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:33.274759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:33.327618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e2180-a41d-4697-91d0-b63bc3793c72 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.359348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 287/578] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:33.360960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.369402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.370236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.370641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.371102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.372085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] RBAC: Adding query filter params (name=network) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:33.372517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.378835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.388355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 292/579] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/services?name=network => generated 124 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:33.394486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.566137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748c7e9-0203-48b8-a9c2-d802f9080b15 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.584617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82070c03-7aa1-4970-9a8e-224731d5ddf0 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.619079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:33.637499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.638130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.638720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.638947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.640201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:33.640654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.642549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.644124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.645012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.645736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.646108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.646832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] RBAC: Adding query filter params (type=network) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:33.647239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.648717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.649941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:33.655785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 288/580] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:33.661084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 293/581] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/services?type=network => generated 350 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:33.661506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.667417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.788287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c004c22-9d67-4558-88ac-7f0c97d315f3 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.796650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dd0010-7509-4d62-b5f8-cc6550681df7 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.829017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.829653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.829923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.830183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.830915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.832876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.839996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.840796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.841169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:33.841516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:33.842283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] RBAC: Adding query filter params (interface=public, service_id=4ccd5e6bd0d14062b6ed09c50d60bc0f, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:33.842708 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:33.844120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:33.845162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:33.846044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:33.846809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:33.846880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 289/582] 10.222.0.23 () {56 vars in 1129 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:33.846880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.857400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 294/583] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/endpoints?service_id=4ccd5e6bd0d14062b6ed09c50d60bc0f&interface=public®ion_id=RegionOne => generated 194 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:33.861540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:33.914542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:33.917596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:33.958588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a7f24-1248-4de4-9b42-ee1aae24cf32 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.971053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272fcebc-48e5-41a4-8fce-68755cc5404a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:33.998882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:33.999486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:33.999753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.000041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.000851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=aa015e7713c34f16be4529081c849ccf) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:34.001117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:34.002548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:34.008494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 290/584] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/role_assignments?user.id=aa015e7713c34f16be4529081c849ccf&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:34.012092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:34.012734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.013014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.013278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.013917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:34.014212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:34.015636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17649fd5-913d-4bee-84e4-06701ad44961 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:34.025790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 295/585] 10.222.0.23 () {56 vars in 1124 bytes} [Wed Oct 12 02:17:33 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:34.025997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:34.026173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942ed71-2be4-41d8-bb10-ee7223a69dce None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.026173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942ed71-2be4-41d8-bb10-ee7223a69dce None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.026173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942ed71-2be4-41d8-bb10-ee7223a69dce None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.027988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 296/586] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:34.029376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942ed71-2be4-41d8-bb10-ee7223a69dce None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:34.156455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942ed71-2be4-41d8-bb10-ee7223a69dce None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:34.168265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:34.179979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117d6965-b5f1-477b-8627-23cd1d22539b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:34.198629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:34.199273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.199608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.199875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.200496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:34.200822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:34.202207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4798f9a5-0738-4103-8d0a-a32d4f980a67 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:34.213659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 297/587] 10.222.0.23 () {56 vars in 1124 bytes} [Wed Oct 12 02:17:34 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:34.213659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.213659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.213659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.220163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:34.232264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:34.233585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.233585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.233585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.234028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:34.234644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:34.235390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:34.235839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:34.242203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 291/588] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:34 2022] GET /identity/v3/services => generated 772 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:34.242809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:34.320436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 298/589] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:34.405150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1fcb34-58fe-4b40-b079-42d83e2a3c25 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:34.457366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:34.458229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.458675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.459104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.459947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] RBAC: Adding query filter params (interface=public, service_id=d5505832b4a3489f9042433f85853551, region_id=RegionOne) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:34.460399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:34.462005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:34.470450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 292/590] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:34 2022] GET /identity/v3/endpoints?service_id=d5505832b4a3489f9042433f85853551&interface=public®ion_id=RegionOne => generated 194 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:34.476287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:34.582081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd60585-80db-448f-a626-89144a1c0347 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:34.585489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:34.621544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:34.622454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.622962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.623693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.624478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:34.626076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:34.631813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 299/591] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:34 2022] GET /identity/v3/services => generated 772 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:34.661223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:34.740077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:34.789337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179a539a-fc1b-4791-ae54-0b5cc2ff928b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:34.841347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:34.841985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:34.842303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:34.842571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:34.854420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:34.859742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:34.860683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:34.861566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 293/592] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:17:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:34.866983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:34.972594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db049fff-d90c-432d-b195-be0ba3887f4f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:35.011090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:35.011621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.011916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.012257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.012945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:35.013242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:35.014616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:35.023289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 300/593] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:17:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:35.029483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:35.156327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad65f3d1-28f0-401b-91dc-4c480e534fd9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:35.170440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:35.192869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:35.193721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.194162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.194567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.209558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:35.211176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:35.229206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:35.230374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:35.231545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:35.231545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:35.231545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:35.232954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 294/594] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:35.238589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:35.274717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:35.368470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b253ad2-eae3-46da-9629-a817eeabffd8 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:35.410238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:35.410894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.411171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.411566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.412213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:35.412511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:35.413969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c811ae-7a54-419a-8ca0-89c3c017321a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:35.427146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 301/595] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:35.432880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:35.547171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28efc2ce-1bd8-4ce6-bedb-e003a4e2daf6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:35.560581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e94832-0207-4108-a578-05169df8b75d None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.560895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e94832-0207-4108-a578-05169df8b75d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.561153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e94832-0207-4108-a578-05169df8b75d None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.562739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 302/596] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:35.568011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.568284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.568554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.600291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:35.601505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:35.604103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:35.604753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.605025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.605289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.618977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:35.621302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:35.641010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:35.642067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:35.643071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:35.643617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 295/597] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:17:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:35.652482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:35.695050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:35.754608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 303/598] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:35.813011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c60017-ff81-4883-beb1-8c6823d81b4d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:35.866455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:35.867062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:35.867337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:35.867605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:35.868330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:35.868633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:35.870085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:35.899226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 296/599] 10.222.0.23 () {56 vars in 1149 bytes} [Wed Oct 12 02:17:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:35.904971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:35.937218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.045535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44be99c-d8e7-4149-b764-7a31d4d669ee None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.095471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.096017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.096293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.096568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.109322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.231555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:36.233063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:36.253102 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.273168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.280426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d410fe3-3cb8-4cc3-8243-4cf54cfbe453 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.321426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.322507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.322863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.323208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.335197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:36.340154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:36.341260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:17:36.342235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:36.342865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 297/600] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:36.349021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.412389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 304/601] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:35 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:17:36.417729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.482120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67758e5f-0815-4e0a-b371-64eac5a18d9e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.516795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.517420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.517691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.517955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.518619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:36.518913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:36.520310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:36.537266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 298/602] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:36.542909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.560715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c0a0e-df22-41ec-bad9-76fa4fa2b472 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.563928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.591429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.597791 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.598428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.598712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.598988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.610032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:36.611438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:36.627285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:36.628261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:36.629185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:36.629185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:36.629185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:36.629185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 305/603] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:36.638021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.717943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e67ed2-ec46-48b1-ae14-e3ba241d63db None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.760879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53bd9f3-c280-48fd-86bb-d6fd0fe240af None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.762002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.762714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.763306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.763659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.781167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:36.782820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:36.798452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.799329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.799777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.800227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.801064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c5899d5d-eec6-4d3e-a4e5-c0cbfc874988 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:36.801578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:36.802547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:36.803518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:36.803518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:36.803518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:36.803700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:36.804145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 299/604] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:36.805624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069a258-79c0-4902-a9e7-982be16b17f0 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.805969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069a258-79c0-4902-a9e7-982be16b17f0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.806311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069a258-79c0-4902-a9e7-982be16b17f0 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.806538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e6ed6-6a28-4ad2-98b4-50713a8117b2 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:36.812081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 300/605] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:36.812081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069a258-79c0-4902-a9e7-982be16b17f0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.825407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 306/606] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:36.826990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.827552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.828058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.847148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:36.848858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:36.881081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.885051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:36.932995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 307/607] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:36 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:36.934027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069a258-79c0-4902-a9e7-982be16b17f0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:36.934622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:36.979533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:36.980151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:36.980431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:36.981835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:36.984232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:36.984548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:36.985956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.018069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 301/608] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:37.021096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.147600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9475f22e-a618-4954-9bfe-abb7561cc2ef None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.172632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a5548c-5d2c-4e77-bb55-ff08270dbee1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.183429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.184595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.185049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.185330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.195496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:37.204526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.206816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.214202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.214833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.215630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.215898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-8daa6cbb-b300-4ca8-a60d-0d43ca61a3fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:37.221617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:37.222489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:37.223374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:37.223815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.223891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 308/609] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:37.224961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.230363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877629b8-6b6c-4e39-b114-f1bab8a6b1e3 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.230776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877629b8-6b6c-4e39-b114-f1bab8a6b1e3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.231141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877629b8-6b6c-4e39-b114-f1bab8a6b1e3 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.232623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 309/610] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:37 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:37.233284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877629b8-6b6c-4e39-b114-f1bab8a6b1e3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:37.239080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:37.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 302/611] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:37.243076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.259289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:37.349881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877629b8-6b6c-4e39-b114-f1bab8a6b1e3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.357682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a89b2e-c823-472d-ac1f-2f842b05735e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.383518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.384112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.384390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.384657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.385401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:37.385705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.387200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89366a3-68df-40af-84e4-32e4ad29f77d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.395420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.396355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.396800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.397226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.397994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.399182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 310/612] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:37.399657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.400371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dbb16e-c970-4191-88c7-fdf779a9e852 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.400653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dbb16e-c970-4191-88c7-fdf779a9e852 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.400955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dbb16e-c970-4191-88c7-fdf779a9e852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:37.412002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:37.413116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:37.414146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.416848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 303/613] 10.222.0.23 () {56 vars in 1132 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:37.417988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.421498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f0dbb16e-c970-4191-88c7-fdf779a9e852 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:37.422680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f0dbb16e-c970-4191-88c7-fdf779a9e852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:37.459280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:37.488191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 311/614] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:37.489453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1dd23-2ab9-45cf-b9aa-16583fb81184 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.489735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1dd23-2ab9-45cf-b9aa-16583fb81184 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.490003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1dd23-2ab9-45cf-b9aa-16583fb81184 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.491228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 312/615] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:37.491904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf1dd23-2ab9-45cf-b9aa-16583fb81184 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.504825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:37.527463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f659afe2-378f-4c57-8290-d23724a8dc2f None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.564402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.565468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.566091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.566424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.567152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:37.567456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.569028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.592303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 304/616] 10.222.0.23 () {56 vars in 1135 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:37.592303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.604207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf1dd23-2ab9-45cf-b9aa-16583fb81184 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.644559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.645185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.645458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.645722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.646443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=f8d2a8079f40422c8bdb9b73b391d0c9) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:37.646750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.648241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfb0ef-bca6-4a71-9718-e12f3f2e5efc None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.654215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 313/617] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/role_assignments?user.id=f8d2a8079f40422c8bdb9b73b391d0c9&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:37.655172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.655452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.655719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.675882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:37.677161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:37.715169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:37.718755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50317af-9fa4-415a-83b6-c407901eebae None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.757484 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 314/618] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:37.758357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.761931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:37.763081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.763952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.765781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.765781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.765781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:37.765781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.767760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.774104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 305/619] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:37.776118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.893426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d246be6-bed5-4e36-bbfd-4b9ab247fd9a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.907434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee364cf-5d6f-4283-aa00-690d93365386 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:37.934832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.936297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.936579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.936895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.946536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:37.947269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:37.947613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:37.947950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] PATH_INFO: `/v3/services/network` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:37.948635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:37.950286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:37.959989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:37.961003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:37.962158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 306/620] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/services/network => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:37.963099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:37.993664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.030748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.032508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.036202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.073329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e573ff69-a68d-4dfa-b7ff-fecda89ca7c3 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.108293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.108941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.109238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.109481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.110205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:38.110511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.111899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.120506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 307/621] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:38.122067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.155454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 315/622] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:37 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:17:38.156460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.231965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.246949 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0609c3-eea9-4722-9e22-c4e664869502 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.268166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb858c2-477a-4848-ab4c-453b53f2387e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.289328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.289956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.290231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.290500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.291131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] RBAC: Adding query filter params (name=network) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:38.291751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.291985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.294142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.303815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 308/623] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/services?name=network => generated 124 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:38.303815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.314023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.315156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.315722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.316201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.317135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.318872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:38.329481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:38.330495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 12 02:17:38.331318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:38.334149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 316/624] 10.222.0.23 () {56 vars in 1129 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:38.335116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.436272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142dcf36-24fa-4755-81b4-46419ad6ed27 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.454257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29c0f76-4dbb-4e70-b97f-7f1211be9906 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.485559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.486117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.486390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.486656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.498912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.499305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.500054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.500525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.501028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.501483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.502256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] RBAC: Adding query filter params (type=network) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:38.502761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.505070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.512445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 317/625] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/services?type=network => generated 350 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:38.513550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:38.522085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:38.523056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:38.523939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:38.523939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:38.523939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:38.527910 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 309/626] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:38.528688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.540758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.557379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.634570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4b5226-1cae-410a-9d60-286ec2592dac None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.666767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ee9064-31f6-4bf5-9579-807d69066f4c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.670738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.671802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.672341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.672888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.673579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:38.673947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.675392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.685028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 318/627] 10.222.0.23 () {56 vars in 1124 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:38.686017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.710262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.711463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.711914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.712509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.713697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.716057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.745949 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 310/628] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:17:38 2022] POST /identity/v3/endpoints => generated 337 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:38.747672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.839581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.841903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:38.854801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9e6b60-94bd-4685-a7fb-3a393f82399b None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.903589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f1b02e-a905-4182-9389-e1a807d726c6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:38.937998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.938979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.939465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.939932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.940863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:38.941362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.942939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.949906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:38.949906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:38.949906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:38.949906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:38.950312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:38.950312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:38.950682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:38.957063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 311/629] 10.222.0.23 () {56 vars in 1124 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:38.969768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:38.972046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 319/630] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:38.977860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:39.125782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d6fe61-56eb-47f9-a8dd-7ecdfc4db807 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:39.153138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f6f8f-6689-4e2e-bd44-5d3e122de347 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:39.154952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:39.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:39.171331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:39.171847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:39.172327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:39.173256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:39.174942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:39.198092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 312/631] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:17:38 2022] POST /identity/v3/endpoints => generated 346 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:39.215397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:39.216494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:39.217049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:39.217539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:39.226427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:39.228368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:39.242107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:39.243093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:39.243993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:39.247228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 320/632] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:39.253116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:39.369840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3656f16f-a85c-4a1b-abc1-9439d334d0a9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:39.407540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:39.408493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:39.408804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:39.409078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:39.409771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:39.410077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:39.411497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe55baf-baa0-465e-a23c-f436f60df2e5 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:39.424320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 313/633] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:39.429897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:39.447347 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:39.537378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03bb43-9f4d-484b-996b-530e1f248102 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:39.572773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:39.573769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:39.574323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:39.574787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:39.575708 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=c57115b9feff40629f5081e32ff4bad3) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:39.576217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:39.577889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a96df0f-d8df-4400-af16-1bb0d1fa597f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:39.584182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 321/634] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:39 2022] GET /identity/v3/role_assignments?user.id=c57115b9feff40629f5081e32ff4bad3&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:39.918659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c6f1bd-dd8f-43bf-8196-fcf84101a1d6 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:39.919014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c6f1bd-dd8f-43bf-8196-fcf84101a1d6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:39.919211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c6f1bd-dd8f-43bf-8196-fcf84101a1d6 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:39.920771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 314/635] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:39.926045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e425bf6-b631-4150-ad6e-fe6f8f03110d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:39.926316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e425bf6-b631-4150-ad6e-fe6f8f03110d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:39.926587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e425bf6-b631-4150-ad6e-fe6f8f03110d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:39.946066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-4e425bf6-b631-4150-ad6e-fe6f8f03110d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:39.947251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4e425bf6-b631-4150-ad6e-fe6f8f03110d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:40.016204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 322/636] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:40.290019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c6f1bd-dd8f-43bf-8196-fcf84101a1d6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:40.343764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:40.407925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c6f1bd-dd8f-43bf-8196-fcf84101a1d6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:40.443427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:40.444061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:40.444339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:40.444614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:40.449370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d991ff-5c1b-4793-aee5-9e74b9738a55 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:40.449650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d991ff-5c1b-4793-aee5-9e74b9738a55 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:40.449921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d991ff-5c1b-4793-aee5-9e74b9738a55 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:40.451495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 323/637] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:40.455331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:40.456345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:40.456617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:40.456928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:40.460174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:40.461134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:40.463024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 315/638] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:17:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:40.467737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:40.475655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:40.482666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:40.484110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:40.567258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 324/639] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:40 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:40.591564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:40.594607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0856101a-9d2c-4ae1-bb07-f589aef2be6d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:40.630113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:40.630716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:40.630985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:40.631249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:40.631895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:40.632193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:40.633564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:40.642291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 316/640] 10.222.0.23 () {56 vars in 1115 bytes} [Wed Oct 12 02:17:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:40.647994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:40.751302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc01c4-2fe5-4594-8b46-98d4af3cdc0f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:40.785976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:40.786596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:40.786869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:40.787137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:40.789849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:40.799089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:40.800557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:40.815764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:40.818032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:40.819880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:40.819880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:40.819880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:40.819880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 325/641] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:40.826063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:40.871772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:40.934775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1dea8f-d858-4442-a532-b15ba1a8bf9a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:40.971560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:40.972601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:40.973050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:40.973470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:40.974278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:40.974748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:40.976512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:40.981906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c16509-3b94-46a9-a628-63864ef60725 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:40.994067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 317/642] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:40.995276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:41.019383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:41.020269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.020690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.021141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.021946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:41.022361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:41.024793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:41.048917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 326/643] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:40 2022] POST /identity/v3/services => generated 269 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:17:41.107666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35c768f-4ae6-4ec7-831c-8a23214fca0c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:41.154894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:41.155550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.155828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.156096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.164654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:41.166161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:41.179613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:41.180885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:41.182081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 318/644] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:17:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:41.189820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:41.249239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:41.306717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141bc9b6-791a-4a09-bb2d-d036cdce59a9 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:41.343098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:41.343752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.344029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.344294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.345044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:41.345343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:41.346775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008fe36c-bdcc-4f57-a253-ddb743b4c539 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:41.366027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 327/645] 10.222.0.23 () {56 vars in 1149 bytes} [Wed Oct 12 02:17:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:41.371634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:41.460221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:41.484630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23a17df-9c8e-4ad1-a061-493457206061 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:41.520398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:41.521442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.521942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.522437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.533598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:41.535330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:41.550431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:41.551402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:41.552308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:41.552308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:41.552308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:41.554399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 319/646] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:41.558990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2062af-88f4-4f2e-b942-e41fedb9a8e2 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.558990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2062af-88f4-4f2e-b942-e41fedb9a8e2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.558990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2062af-88f4-4f2e-b942-e41fedb9a8e2 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.559360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 328/647] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:41.560403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:41.565157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.565157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.565157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.587592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:41.588835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:41.679239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 329/648] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:41 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:41.681572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fb5371-53c3-4be1-9504-a693f429f509 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:41.713535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:41.719717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:41.720579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.721148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.721621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.722480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:41.722979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:41.724645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2194488b-f3a4-42a0-92bf-85c2985c4837 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:41.737616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 320/649] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:41.743288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:41.775131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed5d65a-b48b-4743-8a55-3377300597b6 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.775406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed5d65a-b48b-4743-8a55-3377300597b6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.775675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed5d65a-b48b-4743-8a55-3377300597b6 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.778969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 321/650] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:41.782036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:41.783245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.783543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.783920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.801772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:41.803109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:41.860851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79845e86-5518-48a3-8fe1-ae72fd26760f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:41.895130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 322/651] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:41 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:41.904814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:41.905811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:41.906236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:41.906653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:41.913934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:41.915562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:41.927159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:41.928237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:41.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:41.931283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 330/652] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:41.937760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:41.954911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.057619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:42.065065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31c7dc7-facb-4344-8154-828032a9a8cb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.074836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822907a9-ee4d-4f51-8228-1db0b3c355db None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.098334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:42.109830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.110924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.111667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.112091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.113398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.115140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.117556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.118465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.118587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.118983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.119779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:42.120231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.122073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:42.127500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:42.128684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:42.129709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:42.132779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 331/653] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:41 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:42.139348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.144766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 323/654] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:42.148812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.253602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7037d329-e31b-457c-944b-3438b4eb6288 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.254570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d0481f-23f7-4e8e-8afe-2a62bf2bf418 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.289270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.289933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.290207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.290477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/f8d2a8079f40422c8bdb9b73b391d0c9/roles/ae98a642def1482f9ae5ba673fb7d870` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.292638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=f8d2a8079f40422c8bdb9b73b391d0c9, role_id=ae98a642def1482f9ae5ba673fb7d870)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.294137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.296549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.298121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.298328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.298542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.299207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:42.299479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.300864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.308748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 332/655] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:42 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:42.308748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.320731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 324/656] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:17:42 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/f8d2a8079f40422c8bdb9b73b391d0c9/roles/ae98a642def1482f9ae5ba673fb7d870 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:17:42.320731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.346086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:42.371875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:42.441438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e0da0b-4545-4f5c-a8f2-94736dcb4c47 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.465384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1952c7f4-29ff-4fcc-887f-1184b95ffd22 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.488619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.489979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.490344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.490742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.491680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:42.491982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.493660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.504983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 325/657] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:42 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:42.513050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.513707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.513978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.514240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.514829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.516257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:42.526921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:42.528518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:42.529429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:42.532004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 333/658] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:42.534909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.652121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ef1dd9-43ba-418f-a949-1c4d0c68b264 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.655358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:42.691572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.692181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.692459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.692752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.693491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:42.693798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.695176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c352ace-e170-4700-a47d-e6e1898bd864 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.700186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 326/659] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:42.706113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:42.802275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:42.815167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f6e0b-7af4-47fd-b1c9-eff0e8b4bebd None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:42.851102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:42.851714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:42.851991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:42.852263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:42.852907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:42.853209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:42.854596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db76d1b-72a3-4dc3-a43a-8daf33a23343 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:42.859954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 334/660] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:42 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:43.367122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb7a10-c0b8-4e94-80bc-40bc6ea20304 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:43.367491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb7a10-c0b8-4e94-80bc-40bc6ea20304 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:43.367598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb7a10-c0b8-4e94-80bc-40bc6ea20304 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:43.369253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 327/661] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:43.375334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:43.375614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:43.375884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:43.394942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:43.396123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:43.465970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 335/662] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:43 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:43.741908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eb7a10-c0b8-4e94-80bc-40bc6ea20304 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:43.852806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eb7a10-c0b8-4e94-80bc-40bc6ea20304 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:43.887930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:43.888621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:43.888923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:43.889193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:43.889773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:43.891137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:43.900137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:43.901094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:43.902063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 328/663] 10.222.0.23 () {56 vars in 1132 bytes} [Wed Oct 12 02:17:43 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:43.907735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:43.990217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:44.009987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3850e15-e101-48ea-92d4-d26ffea73c2b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:44.044740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:44.045355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.045632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.045901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.046523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:44.046822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:44.048212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:44.053140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 336/664] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:43 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:44.059240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:44.092011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:44.170969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d5af08-0665-4b71-87c2-1fa280d64815 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:44.210186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:44.210801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.211067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.211340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.211962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:44.212259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:44.213649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:44.219442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 329/665] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:44 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:44.225334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:44.336428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd68bd1-d11c-4fe3-9e07-acd1055df728 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:44.371672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:44.372292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.372570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.372865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.373808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] RBAC: Adding query filter params (interface=public, service_id=c23ea9c2fcf34beb8f037cc7a5b2bec0, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:44.374110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:44.375483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6414664e-749d-41b4-8197-d2cbd2a020b9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:44.380907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 337/666] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:44 2022] GET /identity/v3/endpoints?service_id=c23ea9c2fcf34beb8f037cc7a5b2bec0&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:44.387055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:44.488247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:44.496169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8b3e55-cc07-427e-8ea6-22f2cc11e05a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:44.532293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4d9566-3abe-483e-89d4-83f495d3d790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:44.532936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4d9566-3abe-483e-89d4-83f495d3d790 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.533213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4d9566-3abe-483e-89d4-83f495d3d790 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.533481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4d9566-3abe-483e-89d4-83f495d3d790 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.534080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4d9566-3abe-483e-89d4-83f495d3d790 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:44.535442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4d9566-3abe-483e-89d4-83f495d3d790 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:44.540961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8f8839-6ba3-422f-8937-dca609a570de None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.541239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8f8839-6ba3-422f-8937-dca609a570de None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.541430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 330/667] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:44 2022] GET /identity/v3/services => generated 1029 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:17:44.541573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8f8839-6ba3-422f-8937-dca609a570de None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.543622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 338/668] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:44.550573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1fcbe-a928-4ba6-8bad-e8d98b14cfa1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.550871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1fcbe-a928-4ba6-8bad-e8d98b14cfa1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.551114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1fcbe-a928-4ba6-8bad-e8d98b14cfa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.570547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-56c1fcbe-a928-4ba6-8bad-e8d98b14cfa1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:44.571767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-56c1fcbe-a928-4ba6-8bad-e8d98b14cfa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:44.652945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 331/669] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:44 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:44.889548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a384ea-0e52-44d2-a941-d806a435d122 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.889953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a384ea-0e52-44d2-a941-d806a435d122 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.890154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a384ea-0e52-44d2-a941-d806a435d122 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.891683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 339/670] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:44.896895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:44.897346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:44.897437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:44.916815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:44.917332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a384ea-0e52-44d2-a941-d806a435d122 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:44.917969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:44.931621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:44.996844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 332/671] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:44 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:45.048989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a384ea-0e52-44d2-a941-d806a435d122 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.084628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f305e667-049d-4b11-ae9a-19439a76f1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.085282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f305e667-049d-4b11-ae9a-19439a76f1ac None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.085552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f305e667-049d-4b11-ae9a-19439a76f1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.085822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f305e667-049d-4b11-ae9a-19439a76f1ac None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.096360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f305e667-049d-4b11-ae9a-19439a76f1ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-f305e667-049d-4b11-ae9a-19439a76f1ac None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:17:45.101116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:45.102292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:45.103173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 340/672] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:17:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:45.108981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:45.141160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:45.147066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45718c8c-7702-4abc-b70d-4ba9785b44bd None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.147378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45718c8c-7702-4abc-b70d-4ba9785b44bd None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.147628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45718c8c-7702-4abc-b70d-4ba9785b44bd None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.149840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 341/673] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:45.155965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.156233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.156506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.176224 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:45.177358 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:45.214472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448fb198-b0d1-431b-b84e-2dadbf1c804d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.246799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:45.250877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.251498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.251863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.252134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.252825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:45.253124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:45.260190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:45.260780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 342/674] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:45 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:45.263762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 333/675] 10.222.0.23 () {56 vars in 1115 bytes} [Wed Oct 12 02:17:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:45.269405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:45.285425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:45.376116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:45.396276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74de387e-5d84-4023-a054-b4f92b893b10 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.401097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84d84c8-650f-40c8-8f15-f1f5bf931121 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.432156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.432920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.433839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.434210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.434979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:45.436453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:45.446732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.447424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.449030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.450034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.461087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 334/676] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:45 2022] POST /identity/v3/services => generated 251 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:17:45.464294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:45.466822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:45.485089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:45.486128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:45.487064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:45.487064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:45.487064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:45.489429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 343/677] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:45.497842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:45.550537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:45.611244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:45.634306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05306419-0276-4d3d-b1ac-9d672ad68203 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.670795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5304be81-5ecc-42c6-acab-248fac4fd641 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.679654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.680594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.681145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.681616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.682533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:45.683039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:45.685192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:45.692931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:45.699657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 335/678] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:45.706671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:45.716787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.717411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.717684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.717950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.718599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:45.719968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:45.740783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 344/679] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:45 2022] POST /identity/v3/services => generated 241 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:17:45.838609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372b4e6-018c-4fea-be25-6c0e8104e931 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:45.882222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:45.882862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:45.883139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:45.883402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:45.891881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:45.893384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:45.906241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:45.907687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:45.908634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 336/680] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:17:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:45.915855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:46.027280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0e5150-07c0-484d-a982-94d07381e516 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:46.040128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:46.062646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:46.063249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.063519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.063792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.064507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:46.064846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:46.066279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:46.086803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 345/681] 10.222.0.23 () {56 vars in 1149 bytes} [Wed Oct 12 02:17:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:46.093317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:46.151733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:46.204736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3023b97-bcd1-4308-a5b5-beb696fb0f6c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:46.239403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:46.240013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.240282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.240542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.310012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:46.311371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:46.366204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:46.416645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 337/682] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:17:46 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:17:46.425778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:46.529431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7284b14-10d7-4994-b696-92eaacebf4eb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:46.566031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:46.566984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.567478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.567942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.579333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:46.581067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-336fe986-1c2c-4fa1-a7d4-2124b6427bb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:17:46.596367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:46.597362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:17:46.598274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:17:46.598274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:17:46.598274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:46.600328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 346/683] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:46.606706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:46.717109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d15dd55-ca53-4d2b-be10-f9bf5a4dae79 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:46.751009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5a067-2f65-4a29-8eb2-4680d65af2c0 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.751337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5a067-2f65-4a29-8eb2-4680d65af2c0 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.751552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5a067-2f65-4a29-8eb2-4680d65af2c0 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.753176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 347/684] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:46.754789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:46.755650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.756178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.756652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.757652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:46.758086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.758374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.758476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:46.758649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.760189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:46.780295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 338/685] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:17:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:46.781631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:46.786164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:46.787415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:46.801949 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:46.813758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:46.878760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 348/686] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:46.904950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8367a0-2de5-47f1-a095-fe366fcfa8ca None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:46.940141 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:46.941427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:46.941846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:46.942291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:46.949395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:46.951038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:46.962609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:46.963653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:46.965308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:46.967217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 339/687] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:46.973149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:47.078277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd85529f-c920-44ff-b6c7-51a370408298 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:47.114797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:47.115677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.116149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.116575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.117441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:47.118421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:47.120025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:47.133987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 349/688] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:17:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:47.140143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:47.146131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:47.163021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:47.231259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:47.260279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90460cd-43f9-40e7-978e-89c61500511b None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:47.297940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a692b9-456f-475a-a76b-846d13c59134 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:47.301119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:47.301975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.302444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.302890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.303779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] RBAC: Adding query filter params (role.id=ae98a642def1482f9ae5ba673fb7d870, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=f8d2a8079f40422c8bdb9b73b391d0c9) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:47.304281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:47.306356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:47.321185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 340/689] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:17:47 2022] GET /identity/v3/role_assignments?user.id=f8d2a8079f40422c8bdb9b73b391d0c9&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=ae98a642def1482f9ae5ba673fb7d870 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:47.345976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:47.346694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.346981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.347252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.347850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:47.349296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:47.360012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:47.361054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:47.361958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 350/690] 10.222.0.23 () {56 vars in 1132 bytes} [Wed Oct 12 02:17:47 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:47.367544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:47.485828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e8f2e-751f-42a0-988a-7600b1e94e2a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:47.526093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:47.527132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.527410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.527675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.528292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:47.528587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:47.530032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:47.535156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 341/691] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:47 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:47.541157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:47.627320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:47.654320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9774477a-84e7-4f32-a60a-7f7acb6d311f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:47.693529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:47.694147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.694423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.694692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.695318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:47.695676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:47.697157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f45d5db-294e-488e-93d2-9bfb7dd8ba27 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:47.702803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 351/692] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:47 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:47.708647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:47.711716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:47.771205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de1e91d-4cc3-437a-8b6e-78c9d24894b8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.771379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de1e91d-4cc3-437a-8b6e-78c9d24894b8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.771653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de1e91d-4cc3-437a-8b6e-78c9d24894b8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.774044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 352/693] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:47.781243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c064e33-8830-49d3-9bdb-f42741b693e8 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.781771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c064e33-8830-49d3-9bdb-f42741b693e8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.782262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c064e33-8830-49d3-9bdb-f42741b693e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.800580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5c064e33-8830-49d3-9bdb-f42741b693e8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:47.802150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5c064e33-8830-49d3-9bdb-f42741b693e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:47.823719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148824e1-9d6c-4984-beae-2224f116be6d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:47.864716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a82f5b-1951-4ffb-a786-f0106c299b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:47.865332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a82f5b-1951-4ffb-a786-f0106c299b38 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:47.865632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a82f5b-1951-4ffb-a786-f0106c299b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:47.865883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a82f5b-1951-4ffb-a786-f0106c299b38 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:47.866601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a82f5b-1951-4ffb-a786-f0106c299b38 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:47.868016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a82f5b-1951-4ffb-a786-f0106c299b38 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:47.876974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 353/694] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:47.893887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 342/695] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:17:47 2022] POST /identity/v3/endpoints => generated 347 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:48.079433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe403c9-757b-4f18-bf86-f2394a434ea7 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.079927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe403c9-757b-4f18-bf86-f2394a434ea7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.079927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe403c9-757b-4f18-bf86-f2394a434ea7 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.081540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 354/696] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:48.086849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.087277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.087455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.108505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:48.109855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:48.150612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe403c9-757b-4f18-bf86-f2394a434ea7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.173892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:48.200814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 343/697] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:48.229159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:48.264617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe403c9-757b-4f18-bf86-f2394a434ea7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.300290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.301007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.301344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.301618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.302207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.303611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:48.313884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:48.314946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:48.315778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 355/698] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:48.321200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.426560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebbcde6-4512-4620-b365-22448c607c48 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.460816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.461424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.461699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.461962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.462579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:48.462873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.464224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.474274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 344/699] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:48.475590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.499294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.553781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:48.603456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33e2225-3a3a-40d4-9363-8c7a08cb9c3f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.616646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7e4bb3-3060-47f4-85cb-2d8124ef95d9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.645030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.645908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.646188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.646705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.647740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:48.648044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.649997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.652636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:48.658586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.659453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.659876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.660289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.661049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.661875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 356/700] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:48.662741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.663128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:48.672956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:48.674303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:48.675390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:48.677742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 345/701] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:48.684232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.779561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2310a32b-1474-4827-b422-a124a124a4e4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.788594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d03e5-ff5d-49eb-b08d-aa2a26ea403e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.802502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:48.816449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.817353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.817859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.818323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.819197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] RBAC: Adding query filter params (interface=public, service_id=0b8607fca1fb4bbfa357aeaf4088e58a, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:48.819703 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.821312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.825744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.826394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.826612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.826877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.827490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:48.827781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.829219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 357/702] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/endpoints?service_id=0b8607fca1fb4bbfa357aeaf4088e58a&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:48.833060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.834762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.840180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 346/703] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:48.845761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:48.903919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:48.944064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bde9594-4ce0-410b-b4ca-eb6250228655 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.962284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70993e8-22cb-44cd-b26f-7544c15069ea None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:48.978818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:48.979417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:48.979700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:48.979969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:48.981120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:48.982505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:48.993488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 358/704] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services => generated 1497 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:17:49.008027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:49.008740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:49.009030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:49.009394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:49.010091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:49.010393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:49.012356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:49.018400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 347/705] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:49.024440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:49.140412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b7be0-2619-4aaf-a871-90ddf705b598 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:49.169922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:49.179638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:49.180407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:49.181006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:49.181381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:49.182205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] RBAC: Adding query filter params (interface=public, service_id=e6e06f072bd54819a9921583d7d13ce0, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:49.182634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:49.185106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f30a3-cc7d-47d7-9a1f-eb3ba3f34667 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:49.190678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 359/706] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:49 2022] GET /identity/v3/endpoints?service_id=e6e06f072bd54819a9921583d7d13ce0&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:49.196611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:49.242053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:49.302134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7766b93-8de2-4a3a-9a30-6ac29a735090 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:49.337072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe3a53-9193-4e7d-abb5-982622a3573c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:49.337684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe3a53-9193-4e7d-abb5-982622a3573c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:49.337959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe3a53-9193-4e7d-abb5-982622a3573c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:49.338229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe3a53-9193-4e7d-abb5-982622a3573c None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:49.338825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe3a53-9193-4e7d-abb5-982622a3573c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:49.340691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe3a53-9193-4e7d-abb5-982622a3573c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:49.346952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 348/707] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:49 2022] GET /identity/v3/services => generated 1497 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:17:49.657929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b31efb-9560-44af-8ee9-42e971f24d7f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:49.658592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b31efb-9560-44af-8ee9-42e971f24d7f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:49.659071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b31efb-9560-44af-8ee9-42e971f24d7f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:49.660910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 360/708] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:49.666321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49fab78-e7c3-4afa-9536-0225e4b87a1a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:49.666881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49fab78-e7c3-4afa-9536-0225e4b87a1a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:49.667352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49fab78-e7c3-4afa-9536-0225e4b87a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:49.687451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a49fab78-e7c3-4afa-9536-0225e4b87a1a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:49.689005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a49fab78-e7c3-4afa-9536-0225e4b87a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:49.759035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 349/709] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:50.025751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b31efb-9560-44af-8ee9-42e971f24d7f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:50.141044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b31efb-9560-44af-8ee9-42e971f24d7f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:50.176251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:50.176907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.177186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.177459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] PATH_INFO: `/v3/services/image` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.178036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:50.179513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-bc0011ad-0c7a-49f2-bbab-88aeb015ca71 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:50.189039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:50.190054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:50.191399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:50.192756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0fc85f-de36-4e57-898f-9e2172a9569a None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.192912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 361/710] 10.222.0.23 () {56 vars in 1105 bytes} [Wed Oct 12 02:17:50 2022] GET /identity/v3/services/image => generated 86 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:50.193063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0fc85f-de36-4e57-898f-9e2172a9569a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.193063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0fc85f-de36-4e57-898f-9e2172a9569a None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.193063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 350/711] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:50.197718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.198003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.198275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.200213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0fc85f-de36-4e57-898f-9e2172a9569a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:50.216225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:50.217454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:50.238711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:50.271401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:50.296863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 362/712] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:50.323091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0fc85f-de36-4e57-898f-9e2172a9569a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:50.358485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:50.359114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.359390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.359653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.360268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] RBAC: Adding query filter params (name=image) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:50.360563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:50.362103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:50.367179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 351/713] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:50 2022] GET /identity/v3/services?name=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:50.373258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:50.476498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33b4acc-50b0-4832-a7d2-a5459f60730f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:50.511904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:50.512729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.513145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.513551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.514385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] RBAC: Adding query filter params (type=image) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:50.514820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:50.516418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:50.524815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 363/714] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:50 2022] GET /identity/v3/services?type=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:50.591853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:50.700936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:50.705014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83c66fe-9d8e-4506-a89e-668b4a0e6475 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:50.741243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:50.742233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.742729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.743209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.743992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:50.746181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:50.755645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:50.756602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:17:50.757479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:50.760164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 352/715] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:50 2022] GET /identity/v3/services/compute => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:50.765743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:50.824668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:50.878669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309079e-3a69-4e97-85ff-2e9177dc1574 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:50.913833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:50.914447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:50.914727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:50.914997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:50.915628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:50.915927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:50.917367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d03273-96ab-490d-a0ef-bbe79bb7805e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:50.922841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 364/716] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:50 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:50.928421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:51.036496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7199c524-41ab-42b7-84f5-35aca1b8c42e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:51.070155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:51.070767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:51.071037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:51.071301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:51.071922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] RBAC: Adding query filter params (type=compute) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:51.072218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:51.073608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4ee167-69e0-4ea8-aeae-edace399449a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:51.079145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 353/717] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:50 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:51.314490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30408802-de1c-48b5-9325-fe24ff6be47b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:51.315374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30408802-de1c-48b5-9325-fe24ff6be47b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:51.315788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30408802-de1c-48b5-9325-fe24ff6be47b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:51.317538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 365/718] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:51.323136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9b8f9f-a614-4ade-9247-1e4a2f7b535c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:51.324454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9b8f9f-a614-4ade-9247-1e4a2f7b535c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:51.324976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9b8f9f-a614-4ade-9247-1e4a2f7b535c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:51.343944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-9d9b8f9f-a614-4ade-9247-1e4a2f7b535c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:51.345464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9d9b8f9f-a614-4ade-9247-1e4a2f7b535c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:51.367180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:51.421055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 354/719] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:51.554117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d01745-a66e-441c-b68a-f27a1ac1833b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:51.554785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d01745-a66e-441c-b68a-f27a1ac1833b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:51.555251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d01745-a66e-441c-b68a-f27a1ac1833b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:51.557130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 366/720] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:51.562554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:51.562839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:51.563113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:51.581077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:51.582275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:51.648305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 355/721] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:51.701580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d01745-a66e-441c-b68a-f27a1ac1833b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:51.811618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d01745-a66e-441c-b68a-f27a1ac1833b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:51.825822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:51.848034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:51.848587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:51.848893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:51.849155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:51.849737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:51.851115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:51.860650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:51.861651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:51.862701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 367/722] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:51 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:51.869819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:51.920179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:51.961904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:51.977494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49bca1f-8147-43be-8c64-1e2f139c8bb3 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.012559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.013344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.013704 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.014086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.014801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:52.015191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.016820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.022064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 356/723] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:51 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:52.026494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3925d9cf-1981-4487-929b-51ff299fdc85 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.027530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:52.085874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.086506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.087280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.087563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.088288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.090204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:52.101421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:52.102432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:52.103273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 368/724] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:52.111048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:52.146604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2faf57b-cc6d-4344-8a31-b79cdc1b435d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.181061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:52.181382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.182171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.182281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.182507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.183662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:52.184028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.185669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.195448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 357/725] 10.222.0.23 () {56 vars in 1114 bytes} [Wed Oct 12 02:17:52 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:52.196724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:52.218097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:52.237557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e97ccc-0c0d-45f1-8748-b7568f67d74d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.275588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.276200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.276529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.277355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.278023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:52.278346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.279744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.289312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 369/726] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:52 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:52.291173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:52.331509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767437ef-10bc-411b-894a-6397be9f04cc None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.383275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.383800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.384070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.384332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.385019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.386685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.414286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 358/727] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:17:52 2022] POST /identity/v3/endpoints => generated 346 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:52.428894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aef98e5-8309-4c4e-bf2d-e62648e567cc None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.431750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:52.480224 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.480871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.481188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.481455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.482128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:52.482428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.483838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cf1e19-340e-4e8b-b95a-95bd9aa9cb56 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.491449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 370/728] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:52 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:52.497877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:52.576481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:52.611771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ba1895-60f1-4491-bf57-0c4f5a7714ab None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:52.647201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f310619-7b7b-480d-8e91-b178bc0427f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:52.647734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f310619-7b7b-480d-8e91-b178bc0427f2 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.648007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f310619-7b7b-480d-8e91-b178bc0427f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.648272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f310619-7b7b-480d-8e91-b178bc0427f2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.649026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f310619-7b7b-480d-8e91-b178bc0427f2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:52.650483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f310619-7b7b-480d-8e91-b178bc0427f2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:52.674398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 359/729] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:17:52 2022] POST /identity/v3/endpoints => generated 349 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:52.790736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c1918-feda-4163-aa59-937da03c3434 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.791077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c1918-feda-4163-aa59-937da03c3434 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.791282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c1918-feda-4163-aa59-937da03c3434 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.793525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 371/730] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:52.799440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0155729-281b-4438-97ce-56a54ec26ce2 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:52.800070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0155729-281b-4438-97ce-56a54ec26ce2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:52.800542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0155729-281b-4438-97ce-56a54ec26ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:52.821874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f0155729-281b-4438-97ce-56a54ec26ce2 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:52.823489 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f0155729-281b-4438-97ce-56a54ec26ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:52.906064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 360/731] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:53.178609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c1918-feda-4163-aa59-937da03c3434 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:53.290967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c1918-feda-4163-aa59-937da03c3434 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:53.326467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f0b8f-6971-424d-a761-ef81b003d94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:53.327068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f0b8f-6971-424d-a761-ef81b003d94b None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:53.327339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f0b8f-6971-424d-a761-ef81b003d94b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:53.327603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f0b8f-6971-424d-a761-ef81b003d94b None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:53.328254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18f0b8f-6971-424d-a761-ef81b003d94b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:53.329641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18f0b8f-6971-424d-a761-ef81b003d94b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:53.346376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 372/732] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:53 2022] POST /identity/v3/services => generated 242 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:53.346802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b12281e-af78-455e-9918-e586bf0498cf None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:53.347234 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b12281e-af78-455e-9918-e586bf0498cf None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:53.347455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b12281e-af78-455e-9918-e586bf0498cf None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:53.348925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 361/733] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:53.354525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb8e82d-7efe-4484-91b7-1b05f6e03d51 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:53.355391 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb8e82d-7efe-4484-91b7-1b05f6e03d51 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:53.355673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb8e82d-7efe-4484-91b7-1b05f6e03d51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:53.377218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-abb8e82d-7efe-4484-91b7-1b05f6e03d51 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:53.378434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-abb8e82d-7efe-4484-91b7-1b05f6e03d51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:53.399957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:53.473433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 373/734] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:53 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:53.744475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b12281e-af78-455e-9918-e586bf0498cf None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:53.799111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:53.849465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b12281e-af78-455e-9918-e586bf0498cf None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:53.885460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d458861-38f9-489d-ae60-2c1d5237f118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:53.886178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d458861-38f9-489d-ae60-2c1d5237f118 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:53.886940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d458861-38f9-489d-ae60-2c1d5237f118 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:53.887203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d458861-38f9-489d-ae60-2c1d5237f118 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:53.887859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d458861-38f9-489d-ae60-2c1d5237f118 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:53.889448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d458861-38f9-489d-ae60-2c1d5237f118 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:53.906868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 362/735] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:17:53 2022] POST /identity/v3/services => generated 242 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:17:54.762449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9d01f-b8de-422e-88c2-0c3a26d99199 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:54.763454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9d01f-b8de-422e-88c2-0c3a26d99199 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:54.763963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9d01f-b8de-422e-88c2-0c3a26d99199 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:54.765788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 374/736] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:54.771730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4084ad-7463-4353-8a9d-6932b60e65cf None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:54.772681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4084ad-7463-4353-8a9d-6932b60e65cf None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:54.773181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4084ad-7463-4353-8a9d-6932b60e65cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:54.794335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-db4084ad-7463-4353-8a9d-6932b60e65cf None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:54.795986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-db4084ad-7463-4353-8a9d-6932b60e65cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:54.874935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 363/737] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:54 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:54.898071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac0d710-38f4-4825-a055-393f5678ad13 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:54.898662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac0d710-38f4-4825-a055-393f5678ad13 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:54.899137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac0d710-38f4-4825-a055-393f5678ad13 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:54.900968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 375/738] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:54.906426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:54.906947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:54.907418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:54.926532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:54.927970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:54.959024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:54.999183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 364/739] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:54 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:55.171863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac0d710-38f4-4825-a055-393f5678ad13 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:55.266547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:55.285300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac0d710-38f4-4825-a055-393f5678ad13 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:55.343246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:55.344348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.344813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.345449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.346683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:55.348796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:55.352206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING keystone.server.flask.common [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 12 02:17:55.375608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.sql.core [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 12 02:17:55.375608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 12 02:17:55.375608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [parameters: {'id': '2b3f0f69a05046b093dadd1e8221202a', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 12 02:17:55.375608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81365) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 12 02:17:55.390534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac3dff-d17d-4953-94b5-0d12e975e2ba None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2b3f0f69a05046b093dadd1e8221202a.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2b3f0f69a05046b093dadd1e8221202a. Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:17:55.398106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application result.read() Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 12 02:17:55.398927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application yield resource Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self.session.rollback() Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self.force_reraise() Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise self.value Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application session.commit() Oct 12 02:17:55.400300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._prepare_impl() Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self.session.flush() Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._flush(objects) Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application compat.raise_( Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application flush_context.execute() Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 12 02:17:55.401458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rec.execute(self) Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 12 02:17:55.403766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application result.read() Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:17:55.406006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [parameters: {'id': '2b3f0f69a05046b093dadd1e8221202a', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.408185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:55.410106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2b3f0f69a05046b093dadd1e8221202a. Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 376/740] 10.222.0.23 () {60 vars in 1140 bytes} [Wed Oct 12 02:17:55 2022] POST /identity/v3/projects => generated 262 bytes in 228 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:55.412973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:55.416328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:55.446112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:55.447126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.447622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.448143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.449101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:55.451718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:55.462841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:55.463956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:55.465350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:55.468464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 365/741] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:55.474468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:55.516188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:55.535840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf647ef7-8ac3-4a3d-8122-462ec79facc5 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:55.572721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:55.573359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.574158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.574442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.581812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:55.583675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-95949ec0-4dfa-43e7-b50a-4e0779c8c504 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:17:55.595997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:55.597238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:17:55.598139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:17:55.598565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50583ed4-23c9-4a0e-a162-f2a394b09298 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:55.600846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 377/742] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:55.602454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c426e44-7650-415f-aff5-258906c318fa None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.602801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c426e44-7650-415f-aff5-258906c318fa None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.603081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c426e44-7650-415f-aff5-258906c318fa None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.608658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 378/743] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:55 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:55.612439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c426e44-7650-415f-aff5-258906c318fa None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:55.637308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:55.637840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.638537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.638812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.639440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:55.639733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:55.641121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4507d65-e267-4d68-92d1-d873e4a8335e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:55.645891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 366/744] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:55 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:55.647173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.647454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.647718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.667222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:55.668397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:55.688758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:55.730138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c426e44-7650-415f-aff5-258906c318fa None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:55.743285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 367/745] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:55 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:55.745764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:55.777127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:55.784891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:55.785518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.785795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.786065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.786739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:55.787041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:55.788859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:55.816865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 379/746] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:55 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:55.871350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299c759-64dd-445e-8c60-ef1e861de57e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:55.909763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:55.910739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:55.911212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:55.911675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:55.912571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:55.913106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:55.914827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:55.921698 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 368/747] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:55 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:55.931925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.035877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.051173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa27bf-a0e7-4492-97c2-d553be113733 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.083187 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:56.093053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.094054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.094341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.094793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.095480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] RBAC: Adding query filter params (interface=admin, service_id=e6e06f072bd54819a9921583d7d13ce0, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:56.095790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.099298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.104794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 380/748] 10.222.0.23 () {56 vars in 1249 bytes} [Wed Oct 12 02:17:55 2022] GET /identity/v3/endpoints?service_id=e6e06f072bd54819a9921583d7d13ce0&interface=admin®ion_id=RegionOne => generated 193 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:56.110038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.156508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:56.168086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd24216-2222-497f-a224-c59eea39a3ec None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.205392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.206039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.206317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.206586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] PATH_INFO: `/v3/services/image` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.207168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.208559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-bbd7077a-1907-4712-b5fe-48e4b5b2fe9b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:56.218014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:56.219080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:56.219916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 369/749] 10.222.0.23 () {56 vars in 1105 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services/image => generated 86 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:56.220741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c86a597-9bce-457a-9daf-47a0c7f30368 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.220741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c86a597-9bce-457a-9daf-47a0c7f30368 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.220868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c86a597-9bce-457a-9daf-47a0c7f30368 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.222228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 370/750] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:56.226921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac382f6-fac7-4904-8382-db8d5d7daf7f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.227039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c86a597-9bce-457a-9daf-47a0c7f30368 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.265402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ac9dab-cc1d-4f58-b9f0-101ce684105a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.266009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ac9dab-cc1d-4f58-b9f0-101ce684105a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.266284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ac9dab-cc1d-4f58-b9f0-101ce684105a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.266550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ac9dab-cc1d-4f58-b9f0-101ce684105a None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.267141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ac9dab-cc1d-4f58-b9f0-101ce684105a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.268512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ac9dab-cc1d-4f58-b9f0-101ce684105a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.284730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 381/751] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services => generated 1957 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:17:56.284730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.284730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.284730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.305851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:56.307457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:56.340393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c86a597-9bce-457a-9daf-47a0c7f30368 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.375932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:56.379828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.380797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.381074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.381339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.381976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] RBAC: Adding query filter params (name=image) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:56.382275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.383655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.391306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 371/752] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services?name=image => generated 122 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:56.397246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.398898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 382/753] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:56 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:56.418341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:56.505242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f2327-ee54-49a7-8388-e5c50dcd392c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.540896 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.541859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.542882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.543354 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.544172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] RBAC: Adding query filter params (type=image) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:56.544690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.546313 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.551879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 372/754] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services?type=image => generated 350 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:56.557514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.662512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f852e30b-2d69-4c72-8a01-46c4e5f54f8e None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.686334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.704090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.704844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.705078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.705386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.707054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] RBAC: Adding query filter params (interface=public, service_id=329c1041cc814b9db7299004dc59dad0, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:56.707362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.708805 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.715118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 383/755] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/endpoints?service_id=329c1041cc814b9db7299004dc59dad0&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:56.720738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.779932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:56.791167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:56.812939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d335af25-57c1-4aa2-8c6d-4deb1d113a88 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.846447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c8dcc-fbe3-4ec4-8455-1b8d12478d2a None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:56.858168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.859156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.859620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.860089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] PATH_INFO: `/v3/services/compute` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.860722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.862148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:56.873574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:56.874625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:56.875473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 373/756] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services/compute => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:56.882689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:56.897421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:56.898069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:56.898639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:56.898965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:56.899875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:56.901312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:56.911003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 384/757] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services => generated 1957 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:17:57.007028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2904813b-2830-4410-bbb0-be0bd245ebaa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:57.046341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:57.047337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:57.047793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:57.048265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:57.049120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:57.049608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:57.051183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:57.057859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 374/758] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:56 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:57.063949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:57.178408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c413df-666f-402f-aa30-d6e6ca4b11ec None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:57.192518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:57.216989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:57.217600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:57.217875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:57.218208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:57.218839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:57.219137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:57.220535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:57.225762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 385/759] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:17:57 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:57.232151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:57.245437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:57.338605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bd527-1745-4ab6-bc83-999c2d9dee72 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:57.373587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:57.374193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:57.374469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:57.377229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:57.377229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] RBAC: Adding query filter params (interface=public, service_id=e262fc46ceee412293ca2c7b49aec234, region_id=RegionOne) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:57.377229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:57.377229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2cb598-289f-4a04-96a5-e50804b7958d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:57.382344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 375/760] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:17:57 2022] GET /identity/v3/endpoints?service_id=e262fc46ceee412293ca2c7b49aec234&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:57.388010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:57.499762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0bf88-ba00-42d6-bb43-8523345b25f4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:57.536107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babe483b-b197-49f1-aaaa-fff6d45e9843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:57.536740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babe483b-b197-49f1-aaaa-fff6d45e9843 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:57.537024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babe483b-b197-49f1-aaaa-fff6d45e9843 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:57.537296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babe483b-b197-49f1-aaaa-fff6d45e9843 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:57.537900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babe483b-b197-49f1-aaaa-fff6d45e9843 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:57.539296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babe483b-b197-49f1-aaaa-fff6d45e9843 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:57.546645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 386/761] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:17:57 2022] GET /identity/v3/services => generated 1957 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:17:58.141184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1879a08-11ea-4c82-a2fc-8edbc98d26a4 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:58.141702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1879a08-11ea-4c82-a2fc-8edbc98d26a4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:58.141702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1879a08-11ea-4c82-a2fc-8edbc98d26a4 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:58.143319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 376/762] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:58.148771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd218e12-2ed4-4698-8404-5c02374423a5 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:58.149240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd218e12-2ed4-4698-8404-5c02374423a5 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:58.149394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd218e12-2ed4-4698-8404-5c02374423a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:58.168647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-fd218e12-2ed4-4698-8404-5c02374423a5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:58.170109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-fd218e12-2ed4-4698-8404-5c02374423a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:58.206694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:58.243956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 387/763] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:58 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:58.519946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1879a08-11ea-4c82-a2fc-8edbc98d26a4 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:58.598262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:58.608692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0175f67e-06bc-4e8b-a15f-9c3a834173bd None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:58.609008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0175f67e-06bc-4e8b-a15f-9c3a834173bd None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:58.609280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0175f67e-06bc-4e8b-a15f-9c3a834173bd None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:58.610768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 388/764] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:17:58.616225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:58.616510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:58.616808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:58.626144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1879a08-11ea-4c82-a2fc-8edbc98d26a4 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:58.635101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:58.636881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:58.664171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:58.665013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:58.665311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:58.665588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:58.666888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:58.668963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:58.671769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING keystone.server.flask.common [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 12 02:17:58.687702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.sql.core [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 12 02:17:58.687702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 12 02:17:58.687702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [parameters: {'user_id': '0c72ac1fca07483fac1aed625898c255', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 12 02:17:58.687702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81364) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:17:58.696476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:17:58.697405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application yield resource Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.rollback() Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.force_reraise() Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise self.value Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application session.commit() Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 12 02:17:58.698887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._prepare_impl() Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.flush() Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._flush(objects) Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application compat.raise_( Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application flush_context.execute() Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rec.execute(self) Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 12 02:17:58.699767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:17:58.700673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:17:58.703812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [parameters: {'user_id': '0c72ac1fca07483fac1aed625898c255', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:58.705648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 12 02:17:58.707471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 377/765] 10.222.0.23 () {60 vars in 1131 bytes} [Wed Oct 12 02:17:58 2022] POST /identity/v3/users => generated 160 bytes in 178 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:58.709809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:58.724738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 389/766] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:58 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:17:58.824830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdecb62-83c1-4402-b44e-45d5575c1ceb None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:58.859696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:58.860642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:58.861139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:58.861626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:58.864803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:58.870248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:58.871969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:17:58.884077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:58.885062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:58.886056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 378/767] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:17:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:17:58.892837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.000510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:59.004506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ef5bc-62da-4fb3-9f59-ceef174983cb None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.023235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.046766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.047797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.048414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.048970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.049892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:59.050420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.052294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c19766-02dd-476b-beaa-d50781ec77ed None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.091888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 390/768] 10.222.0.23 () {56 vars in 1101 bytes} [Wed Oct 12 02:17:58 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:59.144550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced7870-9bec-4a6c-b214-ccdd9c95e770 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.145055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced7870-9bec-4a6c-b214-ccdd9c95e770 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.145422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced7870-9bec-4a6c-b214-ccdd9c95e770 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.151047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 391/769] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:59.153413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.153794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.154160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.155718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a8ea0f-a11e-4165-a78f-9fe0ea518b1e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.173643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:59.175015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:59.198486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.199291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.199635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.199975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.200637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.202817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:59.214601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:59.215753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:59.216578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 379/770] 10.222.0.23 () {56 vars in 1126 bytes} [Wed Oct 12 02:17:59 2022] GET /identity/v3/services/object-store => generated 93 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:59.221806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.263273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 392/771] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:59 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:59.305876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:59.352261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db49df-3e75-4cc9-a6a6-092bfe5ba56a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.386948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.387606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.387873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.388132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.388782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:59.389078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.390439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.398770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 380/772] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:59 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:59.404881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.502277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:59.515158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92dec80-5d9d-4a7e-a2b3-f96cf2333372 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.537621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.551173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.551774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.552040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.552299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.552949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:59.553242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.554620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.564272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 393/773] 10.222.0.23 () {56 vars in 1122 bytes} [Wed Oct 12 02:17:59 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:17:59.565216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.651052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b81861-438b-4e2a-8711-2d84ffbe8334 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.673864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d81a5d-84a3-4eb9-9c6c-42427db36d55 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.676265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:59.688039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.688787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.689147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.689485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] PATH_INFO: `/v3/services/image` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.690136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.691571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:17:59.700957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:17:59.702175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:17:59.705804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 381/774] 10.222.0.23 () {56 vars in 1105 bytes} [Wed Oct 12 02:17:59 2022] GET /identity/v3/services/image => generated 86 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:17:59.708433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:17:59.717313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca7e9ea-f5ad-413d-a8e8-2899cea05590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.717923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca7e9ea-f5ad-413d-a8e8-2899cea05590 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.718197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca7e9ea-f5ad-413d-a8e8-2899cea05590 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.718462 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca7e9ea-f5ad-413d-a8e8-2899cea05590 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.719126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca7e9ea-f5ad-413d-a8e8-2899cea05590 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.720493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca7e9ea-f5ad-413d-a8e8-2899cea05590 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.738385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 394/775] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:17:59 2022] POST /identity/v3/endpoints => generated 325 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:17:59.766137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a50e47-9a4a-4969-a7a0-88e530c47225 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.766472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a50e47-9a4a-4969-a7a0-88e530c47225 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.766714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a50e47-9a4a-4969-a7a0-88e530c47225 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.768907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 395/776] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:17:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:17:59.774804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.775078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.775449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.794194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:17:59.795436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:17:59.803983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:17:59.842534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1d155c-797e-4482-9063-28b7e7005f41 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:17:59.881305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 396/777] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:17:59 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:17:59.885160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:17:59.885944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:17:59.886322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:17:59.886690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:17:59.887433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] RBAC: Adding query filter params (name=image) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:17:59.887834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:17:59.889349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:17:59.895214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 382/778] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:59 2022] GET /identity/v3/services?name=image => generated 122 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:17:59.900824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:00.004577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21c92c8-1869-4c5f-8d12-2963c9d03abe None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:00.039065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:00.039844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:00.040218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:00.040587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:00.041342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] RBAC: Adding query filter params (type=image) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:00.041749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:00.043748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:00.053340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 397/779] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:17:59 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:00.054504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:00.118146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:00.155874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:00.160213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:00.166090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c693c9e-8ba9-4174-9c47-5d87ed6b778e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:00.200394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:00.201034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:00.201308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:00.201575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:00.202239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:00.203578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:00.224583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 383/780] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:18:00 2022] POST /identity/v3/endpoints => generated 327 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:00.287008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d27f26-8d66-4d1a-a521-08818671ec47 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:00.331710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:00.332391 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:00.332722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:00.333027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] PATH_INFO: `/v3/services/compute` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:00.333640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:00.335084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:00.345257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:00.346323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 12 02:18:00.347208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:00.349612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 398/781] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:18:00 2022] GET /identity/v3/services/compute => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:00.352645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:00.465071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2154549-b046-48c2-a947-5a220c591da1 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:00.474306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:00.503434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:00.504059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:00.504337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:00.504607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:00.505267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:00.505566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:00.506985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:00.516537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 384/782] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:18:00 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:00.517756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:00.581056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:00.621687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a82e3c2-90f9-4699-913b-4e125a71973f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:00.656014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:00.656643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:00.656950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:00.657212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:00.657848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:00.658144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:00.659580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7feaf7-b0bc-46e2-9d43-9ecc55ecbe74 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:00.669519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 399/783] 10.222.0.23 () {56 vars in 1112 bytes} [Wed Oct 12 02:18:00 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:00.671917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:00.774213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1abba-dc56-41db-9b1b-a82a01d5371a None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:00.807207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a303467-c46c-4867-98df-1665da064bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:00.807873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a303467-c46c-4867-98df-1665da064bed None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:00.808151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a303467-c46c-4867-98df-1665da064bed None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:00.808422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a303467-c46c-4867-98df-1665da064bed None admin] PATH_INFO: `/v3/endpoints` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:00.809123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a303467-c46c-4867-98df-1665da064bed None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:00.810492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a303467-c46c-4867-98df-1665da064bed None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:00.836823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 385/784] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:18:00 2022] POST /identity/v3/endpoints => generated 334 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:01.923465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053746bb-113a-433b-8595-8047dc3cef0a None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:01.923991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053746bb-113a-433b-8595-8047dc3cef0a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:01.923991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053746bb-113a-433b-8595-8047dc3cef0a None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:01.925369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 400/785] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:01.930979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee7ad7b-0a35-4737-83d7-4658c47ccac9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:01.931423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee7ad7b-0a35-4737-83d7-4658c47ccac9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:01.931504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee7ad7b-0a35-4737-83d7-4658c47ccac9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:01.950193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-7ee7ad7b-0a35-4737-83d7-4658c47ccac9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:01.951687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7ee7ad7b-0a35-4737-83d7-4658c47ccac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:01.975506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:02.028716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 386/786] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:02.295624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053746bb-113a-433b-8595-8047dc3cef0a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:02.405808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053746bb-113a-433b-8595-8047dc3cef0a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:02.417564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a297d-72b4-4087-a12a-73d7734e422e None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:02.417852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a297d-72b4-4087-a12a-73d7734e422e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:02.418128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a297d-72b4-4087-a12a-73d7734e422e None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:02.419605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 387/787] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:02.424467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:02.424771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:02.425044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:02.431744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:02.442991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:02.443618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:02.443887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:02.444146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:02.444953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:02.445264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:02.446216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:02.446703 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:02.448732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 401/788] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:02.453971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:02.519902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 388/789] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:02.560807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e99b66-e75b-4ab5-9c90-5729090bb360 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:02.594496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:02.595094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:02.595365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:02.595630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:02.596269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:02.597736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:02.632760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 402/790] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:18:02 2022] POST /identity/v3/projects => generated 311 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:18:02.794060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:02.881807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:02.898683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a18309-555e-4fa5-958f-338d21bc1d80 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:02.934793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:02.935497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:02.935719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:02.935961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:02.936650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:02.938085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:02.947675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.sql.core [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 12 02:18:02.947675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 12 02:18:02.947675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [parameters: {'id': '4420bc2f26264d30a13f51ca6b2d31a5', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 12 02:18:02.947675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81364) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-700ea866-94fe-422d-b7d2-5c6b2ef048b6 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:18:02.955373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 12 02:18:02.956330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application yield resource Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application next(self.gen) Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.rollback() Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.force_reraise() Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise self.value Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application session.commit() Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 12 02:18:02.957736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._prepare_impl() Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.session.flush() Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._flush(objects) Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application compat.raise_( Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application flush_context.execute() Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rec.execute(self) Oct 12 02:18:02.958636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 12 02:18:02.959498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result = self._query(query) Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application conn.query(q) Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application result.read() Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 12 02:18:02.960280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [parameters: {'id': '4420bc2f26264d30a13f51ca6b2d31a5', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 12 02:18:02.962249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:02.964342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:02.966652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 389/791] 10.222.0.23 () {60 vars in 1137 bytes} [Wed Oct 12 02:18:02 2022] POST /identity/v3/domains => generated 190 bytes in 168 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:02.968491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:03.012076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:03.047729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0f0def-e23e-4fa7-9030-cbc80d2c3270 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.048127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0f0def-e23e-4fa7-9030-cbc80d2c3270 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.048470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0f0def-e23e-4fa7-9030-cbc80d2c3270 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.049929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 390/792] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:18:03.054941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.055300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.055569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.071176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2589c0f8-51f7-4254-aec6-5493fe8f9360 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:03.074588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:03.075785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:03.117066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:03.117772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.118052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.118321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.129811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:03.131264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:18:03.146267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:03.147268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:18:03.148165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:18:03.148165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:03.148165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:03.148165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 403/793] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:03.153429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:03.158736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 391/794] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:18:03.250840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:03.255287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48443ab5-dee2-4977-9eb5-c00020319715 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:03.290117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:03.290718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.290990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.291257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.292420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:03.292736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:03.294112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:03.306008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 404/795] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:18:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:18:03.425739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:03.507917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:03.531730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd4f77c-8fc5-4b2f-abdc-1a076fc5ed88 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:03.571619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:03.572255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.572523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.572809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.582997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:03.587694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:03.589389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:03.590453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 392/796] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:18:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:03.596776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:03.696497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba95e9b5-e79d-467c-9f59-b330914e5aa9 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:03.731498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:03.732098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.732374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.732641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.733314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:03.733610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:03.734997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:03.742734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 405/797] 10.222.0.23 () {56 vars in 1115 bytes} [Wed Oct 12 02:18:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:03.749218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:03.846133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69776b02-4677-420f-a5f0-7d32fa8619aa None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:03.881188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:03.881816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:03.882091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:03.882357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:03.889276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:03.893499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:03.894948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:18:03.909873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:03.910824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:18:03.911703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:18:03.911703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:03.911703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:03.911703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 393/798] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:03.916715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:03.938804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:04.017166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c38c259-9071-470c-a555-d7ade90ec253 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:04.049749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:04.050372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.050642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.050907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.051530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:04.051824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:04.053218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:04.065019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 406/799] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:18:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:04.070520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:04.171820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df85dd-d35e-4cf9-afdf-1d69b0313de9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:04.203050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:04.203660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.203930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.204217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.211309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:04.212736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:18:04.223969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:04.225104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:04.226032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 394/800] 10.222.0.23 () {56 vars in 1093 bytes} [Wed Oct 12 02:18:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:04.230884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:04.311752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:04.331495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7d7d9a-8596-4d26-9519-5945484a8351 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:04.364069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:04.364666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.364975 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.365236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.365942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:04.366238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:04.367661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:04.385165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 407/801] 10.222.0.23 () {56 vars in 1133 bytes} [Wed Oct 12 02:18:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:04.391439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:04.440056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:04.488792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7013cec2-aa64-4a30-b1f9-965ae04f7388 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:04.521393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:04.522015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.522292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.522563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.532683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:04.534128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:18:04.548327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:04.549482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:18:04.550368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:18:04.550368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:04.550368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:04.550368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 395/802] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:04.555080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:04.652843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5686d82b-1e7c-408f-b0bc-55f56fa69cdf None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:04.686853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:04.687453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.687730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.688000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.688639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:04.688966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:04.690338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92d9e2-7535-4c82-a8a6-89695513ac31 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:04.696757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:04.702252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 408/803] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:18:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:18:04.708540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:04.808499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d27112-8a80-4b87-8530-363a926a7f54 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:04.810049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:04.830260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c540364-8957-4563-ae84-9c3fe7535b2c None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.830543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c540364-8957-4563-ae84-9c3fe7535b2c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.830811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c540364-8957-4563-ae84-9c3fe7535b2c None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.832129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 409/804] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:18:04.837089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.837366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.837638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.843848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:04.844482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:04.844802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:04.845078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:04.851663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:04.853131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:04.855768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:04.856978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:18:04.864190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:04.865114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:04.866192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 396/805] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:18:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:04.871448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:04.923754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 410/806] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:18:04.969583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd29134-73e7-48ac-8156-82f3d5e003f3 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:05.002099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:05.002696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:05.002973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:05.003241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:05.003918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:05.004219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:05.005661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:05.017345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 397/807] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:18:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:18:05.023172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:05.122166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66eba1-0bce-49e9-845e-7aea2f286c54 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:05.149395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:05.158215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:05.158855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:05.159128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:05.159395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/aa015e7713c34f16be4529081c849ccf/roles/ae98a642def1482f9ae5ba673fb7d870` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:05.161485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=aa015e7713c34f16be4529081c849ccf, role_id=ae98a642def1482f9ae5ba673fb7d870)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:05.162923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:05.176391 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 411/808] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:18:05 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/aa015e7713c34f16be4529081c849ccf/roles/ae98a642def1482f9ae5ba673fb7d870 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:18:05.196546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:05.222997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:05.305131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e60b0a-68bb-4326-becc-0b12bd4a4e17 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:05.341577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbec42af-dc33-4dc7-bc2e-65162e409074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:05.342220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec42af-dc33-4dc7-bc2e-65162e409074 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:05.342497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec42af-dc33-4dc7-bc2e-65162e409074 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:05.342766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec42af-dc33-4dc7-bc2e-65162e409074 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:05.343889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbec42af-dc33-4dc7-bc2e-65162e409074 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:05.345339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbec42af-dc33-4dc7-bc2e-65162e409074 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:05.346921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 398/809] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:05.352907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:05.451736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104ed1e4-3f15-449f-b988-7be0c117bc92 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:05.484618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4d82de-c077-495a-bcfb-3c80cac913ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:05.485243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4d82de-c077-495a-bcfb-3c80cac913ec None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:05.485517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4d82de-c077-495a-bcfb-3c80cac913ec None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:05.485788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4d82de-c077-495a-bcfb-3c80cac913ec None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:05.486454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4d82de-c077-495a-bcfb-3c80cac913ec None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:05.487877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4d82de-c077-495a-bcfb-3c80cac913ec None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:05.509987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 412/810] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:18:05 2022] POST /identity/v3/users => generated 285 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:07.316153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05743a0-3a9e-41dc-bc13-03a9a857367f None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:07.316671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05743a0-3a9e-41dc-bc13-03a9a857367f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:07.316892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05743a0-3a9e-41dc-bc13-03a9a857367f None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:07.318166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 399/811] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:07.323522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35dc814-1537-4d36-956a-50d120d22a13 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:07.323947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35dc814-1537-4d36-956a-50d120d22a13 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:07.324068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35dc814-1537-4d36-956a-50d120d22a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:07.342488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a35dc814-1537-4d36-956a-50d120d22a13 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:07.343678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a35dc814-1537-4d36-956a-50d120d22a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:07.382865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:07.412685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 413/812] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:07.654406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15621ffb-aef6-4774-8c60-c4d887503bb9 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:07.654606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15621ffb-aef6-4774-8c60-c4d887503bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:07.654857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15621ffb-aef6-4774-8c60-c4d887503bb9 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:07.656199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 400/813] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:07.661518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:07.661805 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:07.662078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:07.675746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15621ffb-aef6-4774-8c60-c4d887503bb9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:07.680881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:07.682055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:07.753496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 414/814] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:07.767782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:07.778419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15621ffb-aef6-4774-8c60-c4d887503bb9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:07.811121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:07.811734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:07.812005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:07.812268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:07.821775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:07.826491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:07.827639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:07.828627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 401/815] 10.222.0.23 () {56 vars in 1120 bytes} [Wed Oct 12 02:18:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:07.833650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:07.929034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6631409d-1ce7-4a13-97ab-5dad6b3362f8 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:07.931181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:07.963440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:07.964036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:07.964305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:07.964568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:07.965243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:07.965540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:07.966928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:07.974512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 415/816] 10.222.0.23 () {56 vars in 1115 bytes} [Wed Oct 12 02:18:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:07.980307 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.016460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.079219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939c0c1b-794b-4892-a333-ea469e40e1ae None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.112177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.113082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.113353 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.113614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.115843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed26edd1-52d3-404e-b596-9c481f5034d3 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.124228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.125681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:18:08.140510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:08.141525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:18:08.147485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:18:08.147485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:08.147485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:08.147485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 402/817] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:08.147485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.150122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.151410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.152204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.152485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.152777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.163379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:08.168404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:08.169626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:08.170476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 416/818] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:08.175551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.204470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.248133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd4f2e-9f98-471d-bd3f-7221611c65f6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.276639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d13282-e53f-4a32-ab2f-1a26d9ebf693 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.282541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.283159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.283855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.284132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.284803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:08.285108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.286490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.299199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 403/819] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:08.304603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.314520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.315129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.315403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.315670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.316325 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:08.316628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.318029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.326698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 417/820] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:08.332362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.366645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.405511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ed94ec-bd61-4a8e-9bef-c1f0b4cee8c8 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.420771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.434326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98a8a5b-7ab9-4969-9cc0-24ebbb21303f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.439554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.440181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.440455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.440750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.447961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.449410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:18:08.460536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:08.461570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:08.463043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 404/821] 10.222.0.23 () {56 vars in 1093 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:18:08.467340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.469450 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.470081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.470351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.470623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] PATH_INFO: `/v3/users/1e1411ec129d4a86be2bc32c25a1d60e` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.484685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] RBAC: Authorizing `identity:get_user(user_id=1e1411ec129d4a86be2bc32c25a1d60e)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.486182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.487806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 418/822] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/users/1e1411ec129d4a86be2bc32c25a1d60e => generated 285 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:18:08.493364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.566407 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538ac8c-562f-4e28-ae21-2fc4033b0710 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.588674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.589895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b0493-aabc-4f3f-9b0c-093859fcbd7d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.601140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.601750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.602026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.602296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.603018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:08.603323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.605271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.623437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 405/823] 10.222.0.23 () {56 vars in 1133 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:18:08.624011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.624633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.624935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.625234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] PATH_INFO: `/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.626306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] RBAC: Authorizing `identity:get_project(project_id=9b79ab03c1ef4bfcaa81d7b727621780)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.627728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.629034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.629403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 419/824] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:18:08.634996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:08.649677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.726690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e3bd29-287b-46bf-8089-fbddddfb8d01 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.733551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5fd528-4e70-46f0-b555-c08506d714f1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:08.762605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.763211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.763487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.763755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.766622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:08.767273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:08.767545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:08.767820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] PATH_INFO: `/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780/users/1e1411ec129d4a86be2bc32c25a1d60e/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:08.769736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] RBAC: Authorizing `identity:create_grant(project_id=9b79ab03c1ef4bfcaa81d7b727621780, user_id=1e1411ec129d4a86be2bc32c25a1d60e, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.771178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.784148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 420/825] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:18:08 2022] PUT /identity/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780/users/1e1411ec129d4a86be2bc32c25a1d60e/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:18:08.817472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:08.839889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:08.841402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:08.946470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 406/826] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 12 02:18:08.952320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:09.027092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:09.053039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9f332-158a-4e83-9523-4c8310704f0d None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:09.087774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:09.088397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:09.089285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:09.089550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:09.099952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:09.101554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:18:09.116794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:09.117862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:18:09.123845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:18:09.123845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:18:09.123845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:09.123845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 421/827] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:09.124131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:09.200545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:09.219755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab9a46e-d2b3-4a61-b31d-6b50e7821c95 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:09.252684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:09.253337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:09.253610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:09.253875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:09.254501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:09.254794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:09.256150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:09.267743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 407/828] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:18:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:09.273702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:09.487004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ca57a-e672-4452-99d5-330bbf63354f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:09.512595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:09.521582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:09.522200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:09.522469 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:09.522730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:09.529133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:09.530558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:18:09.541226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:09.542255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:18:09.543355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:09.544974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 422/829] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:18:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:09.548308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:09.646108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf1918-fbb6-4059-a8f9-d0a69089d202 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:09.679031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:09.679641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:09.679918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:09.680185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:09.680901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:09.681227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:09.683137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fc0ad-8a74-4738-a5ba-cb9cc34b3991 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:09.691800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:09.696474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 408/830] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:18:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:09.701837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:09.799217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b993d8-a46a-47a3-9279-88884a21df4b None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:09.831464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:09.832050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:09.832322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:09.832586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:09.833321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] RBAC: Adding query filter params (role.id=ae98a642def1482f9ae5ba673fb7d870, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=aa015e7713c34f16be4529081c849ccf) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:09.833626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:09.835057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729f53dc-50fe-4096-b143-b7899d4815a5 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:09.840683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 423/831] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:18:09 2022] GET /identity/v3/role_assignments?user.id=aa015e7713c34f16be4529081c849ccf&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=ae98a642def1482f9ae5ba673fb7d870 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:10.892016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f04c2ff-5c66-49a8-9a0d-e7de9eed9ca0 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:10.892568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f04c2ff-5c66-49a8-9a0d-e7de9eed9ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:10.892568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f04c2ff-5c66-49a8-9a0d-e7de9eed9ca0 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:10.893913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 409/832] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:10.898945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:10.899326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:10.899488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:10.918239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:10.919406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:10.963982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:10.985395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 424/833] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:11.249747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f04c2ff-5c66-49a8-9a0d-e7de9eed9ca0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:11.347413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f04c2ff-5c66-49a8-9a0d-e7de9eed9ca0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:11.379148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:11.379758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:11.380030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:11.380296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:11.390686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:11.395425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:11.396511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:11.397605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 410/834] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:18:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:11.402949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:11.498029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cc590d-ea20-445b-84bb-59822d175aa3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:11.531507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:11.532044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:11.532336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:11.532611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:11.533299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:11.533599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:11.534966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:11.542746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 425/835] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:18:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:11.548409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:11.606260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:11.646738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633e2193-c5c9-4e83-95d3-d7cdcf06d431 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:11.678887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:11.679500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:11.679778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:11.680048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] PATH_INFO: `/v3/users/1e1411ec129d4a86be2bc32c25a1d60e` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:11.681175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] RBAC: Authorizing `identity:get_user(user_id=1e1411ec129d4a86be2bc32c25a1d60e)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:11.683124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:11.684797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 411/836] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:11 2022] GET /identity/v3/users/1e1411ec129d4a86be2bc32c25a1d60e => generated 285 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:11.690278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:11.733221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:11.788689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3288239-a915-4e7f-a061-7a780eafdace None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:11.821169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:11.821756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:11.822035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:11.822305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:11.888767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:11.890206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:11.929422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:11.985112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 426/837] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:18:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:18:11.990866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:12.086979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48fbef2-497d-496b-8028-d0dc4ed9a567 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:12.115343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:12.124224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285be5fe-edb1-4733-81b8-d69d1c8bd391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:12.124883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285be5fe-edb1-4733-81b8-d69d1c8bd391 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:12.125161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285be5fe-edb1-4733-81b8-d69d1c8bd391 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:12.125429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285be5fe-edb1-4733-81b8-d69d1c8bd391 None admin] PATH_INFO: `/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:12.126519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285be5fe-edb1-4733-81b8-d69d1c8bd391 None admin] RBAC: Authorizing `identity:get_project(project_id=9b79ab03c1ef4bfcaa81d7b727621780)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:12.127960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285be5fe-edb1-4733-81b8-d69d1c8bd391 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:12.129602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 412/838] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:11 2022] GET /identity/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:12.135616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:12.230936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ac5e09-4578-4398-b37f-4e35aeed526b None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:12.264308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:12.264914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:12.265191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:12.265455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:12.266165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=9b79ab03c1ef4bfcaa81d7b727621780, user.id=1e1411ec129d4a86be2bc32c25a1d60e) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:12.266466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:12.267887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99b5ffd-308a-48ce-bcf0-3151b388090e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:12.273688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 427/839] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:18:12 2022] GET /identity/v3/role_assignments?user.id=1e1411ec129d4a86be2bc32c25a1d60e&scope.project.id=9b79ab03c1ef4bfcaa81d7b727621780&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:14.393362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc76499-85d1-4e5a-97a3-36b306818c11 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:14.393867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc76499-85d1-4e5a-97a3-36b306818c11 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:14.393867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc76499-85d1-4e5a-97a3-36b306818c11 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:14.395143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 413/840] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:14.400412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:14.401419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:14.401580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:14.414093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:14.419722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:14.420930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:14.485342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 428/841] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:14.755045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc76499-85d1-4e5a-97a3-36b306818c11 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:14.852554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc76499-85d1-4e5a-97a3-36b306818c11 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:14.885162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c12712b-b003-45c7-a432-ff3fcb64ddad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:14.885770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c12712b-b003-45c7-a432-ff3fcb64ddad None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:14.886045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c12712b-b003-45c7-a432-ff3fcb64ddad None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:14.886311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c12712b-b003-45c7-a432-ff3fcb64ddad None admin] PATH_INFO: `/v3/domains/default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:14.887405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c12712b-b003-45c7-a432-ff3fcb64ddad None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:14.888849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c12712b-b003-45c7-a432-ff3fcb64ddad None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:14.890387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 414/842] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:14.895934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:14.991040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750995c3-b0a7-4207-8629-31e4fd5e8e18 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:15.023919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bdc362-6dcd-4328-8fde-d4dbf4f789de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:15.024457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bdc362-6dcd-4328-8fde-d4dbf4f789de None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:15.024752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bdc362-6dcd-4328-8fde-d4dbf4f789de None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:15.025024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bdc362-6dcd-4328-8fde-d4dbf4f789de None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:15.025767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bdc362-6dcd-4328-8fde-d4dbf4f789de None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:15.027189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bdc362-6dcd-4328-8fde-d4dbf4f789de None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:15.035870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:15.048785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 429/843] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:18:14 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:18:17.164660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc2f94-daa6-4ae9-85e2-f2fe097508e4 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:17.165129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc2f94-daa6-4ae9-85e2-f2fe097508e4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:17.165226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc2f94-daa6-4ae9-85e2-f2fe097508e4 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:17.167113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 415/844] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:17.172232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:17.172623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:17.172803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:17.190519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:17.191698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:17.257029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 430/845] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:17.526688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcc2f94-daa6-4ae9-85e2-f2fe097508e4 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:17.586975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:17.624931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcc2f94-daa6-4ae9-85e2-f2fe097508e4 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:17.657024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:17.657631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:17.657900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:17.658162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] PATH_INFO: `/v3/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:17.659512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] RBAC: Authorizing `identity:get_role(role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:17.661008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:17.662292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 416/846] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:17 2022] GET /identity/v3/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:17.668607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:17.764987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3b23f3-c334-40f9-9ff0-485520d4790e None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:17.787467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:17.799553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:17.800163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:17.800432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:17.800711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] PATH_INFO: `/v3/users/a28498d2c99e425c8c03ab5c1c5d64fe` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:17.813054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] RBAC: Authorizing `identity:get_user(user_id=a28498d2c99e425c8c03ab5c1c5d64fe)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:17.814488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:17.816042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 431/847] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:17 2022] GET /identity/v3/users/a28498d2c99e425c8c03ab5c1c5d64fe => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:17.821758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:17.916777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6137341-1fde-4193-8d66-bcb170ac3f5c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:17.943797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:17.948844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756fcc74-432d-4381-ad09-e3ce29a4572f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:17.949453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756fcc74-432d-4381-ad09-e3ce29a4572f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:17.949732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756fcc74-432d-4381-ad09-e3ce29a4572f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:17.949987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756fcc74-432d-4381-ad09-e3ce29a4572f None admin] PATH_INFO: `/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:17.951071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756fcc74-432d-4381-ad09-e3ce29a4572f None admin] RBAC: Authorizing `identity:get_project(project_id=9b79ab03c1ef4bfcaa81d7b727621780)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:17.952467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756fcc74-432d-4381-ad09-e3ce29a4572f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:17.954066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 417/848] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:17 2022] GET /identity/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:17.959923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:18.055260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf874570-0e02-416a-b3de-5de9869fd6a8 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:18.088603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c51099-a5fd-46f0-b6c4-17a3845331f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:18.089271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c51099-a5fd-46f0-b6c4-17a3845331f9 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:18.089544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c51099-a5fd-46f0-b6c4-17a3845331f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:18.089813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c51099-a5fd-46f0-b6c4-17a3845331f9 None admin] PATH_INFO: `/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780/users/a28498d2c99e425c8c03ab5c1c5d64fe/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:18.091912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c51099-a5fd-46f0-b6c4-17a3845331f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=9b79ab03c1ef4bfcaa81d7b727621780, user_id=a28498d2c99e425c8c03ab5c1c5d64fe, role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:18.093368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c51099-a5fd-46f0-b6c4-17a3845331f9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:18.107263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 432/849] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:18:17 2022] PUT /identity/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780/users/a28498d2c99e425c8c03ab5c1c5d64fe/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:18:20.211596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae916878-1d79-4eb4-b234-5764a646c7d6 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:20.212113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae916878-1d79-4eb4-b234-5764a646c7d6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:20.212233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae916878-1d79-4eb4-b234-5764a646c7d6 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:20.214961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 418/850] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:20.220348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:20.220758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:20.220918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:20.239221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:20.240523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:20.275005 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:20.310653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 433/851] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:20.581093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae916878-1d79-4eb4-b234-5764a646c7d6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:20.682285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae916878-1d79-4eb4-b234-5764a646c7d6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:20.716138 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:20.716788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:20.717067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:20.717336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] PATH_INFO: `/v3/roles/cd7b8279c49f4441b98ba176e7e27199` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:20.718720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] RBAC: Authorizing `identity:get_role(role_id=cd7b8279c49f4441b98ba176e7e27199)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:20.720086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:20.721440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 419/852] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:20 2022] GET /identity/v3/roles/cd7b8279c49f4441b98ba176e7e27199 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:20.727336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:20.824549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88db01b9-a5db-43c4-8f11-353ce8680b9c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:20.858023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:20.858590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:20.858868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:20.859139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] PATH_INFO: `/v3/users/a28498d2c99e425c8c03ab5c1c5d64fe` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:20.860249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] RBAC: Authorizing `identity:get_user(user_id=a28498d2c99e425c8c03ab5c1c5d64fe)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:20.861723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:20.863263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 434/853] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:20 2022] GET /identity/v3/users/a28498d2c99e425c8c03ab5c1c5d64fe => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:20.868900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:20.933384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:20.966051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59805d8-51d5-4734-ac39-380fe03ba991 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:20.998986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:20.999582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:20.999857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:21.000123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:21.063865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:21.065329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:21.127677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:21.161055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 420/854] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:18:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:18:21.167009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:21.205840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:21.264397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6358bfe-7562-4f8b-aca6-3674d11cac81 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:21.297546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361bb14f-65e7-4fa5-85e5-2cb352399c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:21.298155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361bb14f-65e7-4fa5-85e5-2cb352399c8d None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:21.298431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361bb14f-65e7-4fa5-85e5-2cb352399c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:21.298698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361bb14f-65e7-4fa5-85e5-2cb352399c8d None admin] PATH_INFO: `/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:21.299795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361bb14f-65e7-4fa5-85e5-2cb352399c8d None admin] RBAC: Authorizing `identity:get_project(project_id=9b79ab03c1ef4bfcaa81d7b727621780)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:21.301261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361bb14f-65e7-4fa5-85e5-2cb352399c8d None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:21.302858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 435/855] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:21 2022] GET /identity/v3/projects/9b79ab03c1ef4bfcaa81d7b727621780 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:21.309122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:21.404200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19650b8-17a2-43a1-8207-f5e9ead82117 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:21.436480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:21.437094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:21.437369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:21.437635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:21.438343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] RBAC: Adding query filter params (role.id=cd7b8279c49f4441b98ba176e7e27199, scope.project.id=9b79ab03c1ef4bfcaa81d7b727621780, user.id=a28498d2c99e425c8c03ab5c1c5d64fe) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:21.438646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:21.440047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9306e1-baee-4c33-bd9d-9ed8b2a09454 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:21.445562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 421/856] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:18:21 2022] GET /identity/v3/role_assignments?user.id=a28498d2c99e425c8c03ab5c1c5d64fe&scope.project.id=9b79ab03c1ef4bfcaa81d7b727621780&role.id=cd7b8279c49f4441b98ba176e7e27199 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:23.559912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b6b605-fe63-490b-bed8-0ffae007a6f5 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:23.560460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b6b605-fe63-490b-bed8-0ffae007a6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:23.560460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b6b605-fe63-490b-bed8-0ffae007a6f5 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:23.561806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 436/857] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:23.566968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:23.567334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:23.567514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:23.585301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:23.586496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:23.611647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:23.652497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 422/858] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:23.916959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b6b605-fe63-490b-bed8-0ffae007a6f5 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:24.015383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b6b605-fe63-490b-bed8-0ffae007a6f5 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:24.022360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:24.048814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f803f228-dfb7-44f4-ab53-75063f34deac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:24.049436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f803f228-dfb7-44f4-ab53-75063f34deac None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:24.049714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f803f228-dfb7-44f4-ab53-75063f34deac None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:24.049983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f803f228-dfb7-44f4-ab53-75063f34deac None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:24.051096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f803f228-dfb7-44f4-ab53-75063f34deac None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:24.052546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f803f228-dfb7-44f4-ab53-75063f34deac None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:24.054126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 437/859] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:24.059889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:24.154201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468de18d-2f01-4f03-a77e-2ace2145e927 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:24.185681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d41b45-ef3c-4223-ae1e-4ac74c9cebb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:24.186214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d41b45-ef3c-4223-ae1e-4ac74c9cebb2 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:24.186483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d41b45-ef3c-4223-ae1e-4ac74c9cebb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:24.186748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d41b45-ef3c-4223-ae1e-4ac74c9cebb2 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:24.187401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d41b45-ef3c-4223-ae1e-4ac74c9cebb2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:24.188825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d41b45-ef3c-4223-ae1e-4ac74c9cebb2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:24.214658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 423/860] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:18:24 2022] POST /identity/v3/projects => generated 311 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:26.336103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c81dbbd-ea51-4499-91ff-8a19b590f64d None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:26.336753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c81dbbd-ea51-4499-91ff-8a19b590f64d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:26.336753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c81dbbd-ea51-4499-91ff-8a19b590f64d None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:26.338085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 438/861] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:26.343288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:26.343682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:26.343824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:26.362036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:26.363196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:26.374300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:26.430159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 424/862] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:26.695832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c81dbbd-ea51-4499-91ff-8a19b590f64d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:26.796103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c81dbbd-ea51-4499-91ff-8a19b590f64d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:26.828094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4617a355-338e-4687-88db-05016bacdcb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:26.828728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4617a355-338e-4687-88db-05016bacdcb7 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:26.829001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4617a355-338e-4687-88db-05016bacdcb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:26.829269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4617a355-338e-4687-88db-05016bacdcb7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:26.830370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4617a355-338e-4687-88db-05016bacdcb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:26.831800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4617a355-338e-4687-88db-05016bacdcb7 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:26.833365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 439/863] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:18:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:26.839522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:26.934213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bb0c1f-d479-4b29-950a-74e4abb6f9a1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:26.966606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894cccad-5c10-4191-a287-3ddf9d12f316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:26.967170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894cccad-5c10-4191-a287-3ddf9d12f316 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:26.967445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894cccad-5c10-4191-a287-3ddf9d12f316 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:26.967708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894cccad-5c10-4191-a287-3ddf9d12f316 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:26.968356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894cccad-5c10-4191-a287-3ddf9d12f316 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:26.969785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894cccad-5c10-4191-a287-3ddf9d12f316 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:26.986257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:26.991324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 425/864] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:18:26 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:18:29.112056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40055a11-da31-4823-a6be-409df1c1049f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:29.112587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40055a11-da31-4823-a6be-409df1c1049f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:29.112587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40055a11-da31-4823-a6be-409df1c1049f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:29.113949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 440/865] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:29.119130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:29.119564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:29.119663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:29.138192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:29.139348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:29.202080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 426/866] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:29.468064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40055a11-da31-4823-a6be-409df1c1049f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:29.511212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:29.566430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40055a11-da31-4823-a6be-409df1c1049f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:29.599385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:29.599988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:29.600260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:29.600526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:29.610227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:29.614955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:29.616090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:29.617171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 441/867] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:18:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:29.622678 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:29.718318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e38451-cdbc-4663-86a4-a278624fc453 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:29.745525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:29.751696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:29.752297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:29.752571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:29.752869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:29.753524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:29.753828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:29.755190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:29.764009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 427/868] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:18:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:29.769846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:29.869973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a722c4-b698-4fe4-b5d7-61a9c2764894 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:29.871486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:29.905428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:29.906048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:29.906320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:29.906586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] PATH_INFO: `/v3/users/919d64211f924b79820ae15d7f4f6f1f` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:29.919290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] RBAC: Authorizing `identity:get_user(user_id=919d64211f924b79820ae15d7f4f6f1f)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:29.920821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:29.922771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 442/869] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:29 2022] GET /identity/v3/users/919d64211f924b79820ae15d7f4f6f1f => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:29.928313 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:30.031833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab79e96e-6467-48be-8c99-adebcf81e1ac None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:30.064337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7f70a-18fe-4cb4-adc5-9b695e1ab416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:30.065025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7f70a-18fe-4cb4-adc5-9b695e1ab416 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:30.065296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7f70a-18fe-4cb4-adc5-9b695e1ab416 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:30.065562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7f70a-18fe-4cb4-adc5-9b695e1ab416 None admin] PATH_INFO: `/v3/projects/6947d53ae26542e99c46f947eb09ec85` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:30.066658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7f70a-18fe-4cb4-adc5-9b695e1ab416 None admin] RBAC: Authorizing `identity:get_project(project_id=6947d53ae26542e99c46f947eb09ec85)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:30.068070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7f70a-18fe-4cb4-adc5-9b695e1ab416 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:30.069738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 428/870] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:29 2022] GET /identity/v3/projects/6947d53ae26542e99c46f947eb09ec85 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:30.075739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:30.172278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09ff732-95a3-4400-b753-7160b53ee766 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:30.205063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e015a767-a363-416c-bc6b-20ecdec7ce55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:30.205704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e015a767-a363-416c-bc6b-20ecdec7ce55 None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:30.205978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e015a767-a363-416c-bc6b-20ecdec7ce55 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:30.206243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e015a767-a363-416c-bc6b-20ecdec7ce55 None admin] PATH_INFO: `/v3/projects/6947d53ae26542e99c46f947eb09ec85/users/919d64211f924b79820ae15d7f4f6f1f/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:30.208093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e015a767-a363-416c-bc6b-20ecdec7ce55 None admin] RBAC: Authorizing `identity:create_grant(project_id=6947d53ae26542e99c46f947eb09ec85, user_id=919d64211f924b79820ae15d7f4f6f1f, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:30.209556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e015a767-a363-416c-bc6b-20ecdec7ce55 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:30.221953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 443/871] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:18:30 2022] PUT /identity/v3/projects/6947d53ae26542e99c46f947eb09ec85/users/919d64211f924b79820ae15d7f4f6f1f/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:18:32.320607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea40cc36-f418-45c1-80d1-0f7543c4e7a8 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:32.321160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea40cc36-f418-45c1-80d1-0f7543c4e7a8 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:32.321160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea40cc36-f418-45c1-80d1-0f7543c4e7a8 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:32.322989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 429/872] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:32.328223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:32.328646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:32.328785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:32.346950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:32.348123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:32.361724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:32.414142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 444/873] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:32.682973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea40cc36-f418-45c1-80d1-0f7543c4e7a8 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:32.742144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:32.783069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea40cc36-f418-45c1-80d1-0f7543c4e7a8 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:32.815337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:32.815947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:32.816219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:32.816485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:32.826746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:32.831473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:32.832554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:32.833782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 430/874] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:18:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:32.839138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:32.935498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cba9a12-b603-429b-ab2a-fa9b15642d60 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:32.967685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:32.968274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:32.968547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:32.968841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:32.969491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:32.969785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:32.971148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:32.979404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 445/875] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:18:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:32.985133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:33.080091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddedef74-4767-424a-8a48-b24e0416886c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:33.099140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:33.112979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:33.113596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:33.113867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:33.114130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] PATH_INFO: `/v3/users/919d64211f924b79820ae15d7f4f6f1f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:33.115225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] RBAC: Authorizing `identity:get_user(user_id=919d64211f924b79820ae15d7f4f6f1f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:33.116640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:33.118220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 431/876] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:32 2022] GET /identity/v3/users/919d64211f924b79820ae15d7f4f6f1f => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:33.124310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:33.139690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:33.222251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41251f3-af6d-41a9-b9ab-ed8a2c0e4d00 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:33.254160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:33.254746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:33.255020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:33.255284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:33.319516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:33.320975 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:33.334235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:33.415977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 446/877] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:18:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:18:33.421747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:33.517760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209f6828-4e52-4d0f-909e-b1b910ed4746 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:33.548881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb534c3-23aa-49b1-84f6-9811c34d761d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:33.549478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb534c3-23aa-49b1-84f6-9811c34d761d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:33.549747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb534c3-23aa-49b1-84f6-9811c34d761d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:33.550011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb534c3-23aa-49b1-84f6-9811c34d761d None admin] PATH_INFO: `/v3/projects/6947d53ae26542e99c46f947eb09ec85` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:33.551092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb534c3-23aa-49b1-84f6-9811c34d761d None admin] RBAC: Authorizing `identity:get_project(project_id=6947d53ae26542e99c46f947eb09ec85)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:33.552505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb534c3-23aa-49b1-84f6-9811c34d761d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:33.554095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 432/878] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:33 2022] GET /identity/v3/projects/6947d53ae26542e99c46f947eb09ec85 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:33.560436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:33.655099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4220015b-7dfc-41bb-b581-7dfa99c30c59 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:33.667487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:33.688114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:33.688724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:33.688991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:33.689253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:33.689953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=6947d53ae26542e99c46f947eb09ec85, user.id=919d64211f924b79820ae15d7f4f6f1f) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:33.690250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:33.691651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d1ad6-b7ef-4730-81f6-9861c77b4b39 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:33.697706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 447/879] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:18:33 2022] GET /identity/v3/role_assignments?user.id=919d64211f924b79820ae15d7f4f6f1f&scope.project.id=6947d53ae26542e99c46f947eb09ec85&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:35.808107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929de2c9-97cb-4f6b-963f-8cff30ea39f1 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:35.808654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929de2c9-97cb-4f6b-963f-8cff30ea39f1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:35.808654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929de2c9-97cb-4f6b-963f-8cff30ea39f1 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:35.810154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 433/880] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:35.815237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5c3d2-9698-46cf-b98f-7c82eb311511 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:35.815602 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5c3d2-9698-46cf-b98f-7c82eb311511 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:35.815770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5c3d2-9698-46cf-b98f-7c82eb311511 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:35.834067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-dea5c3d2-9698-46cf-b98f-7c82eb311511 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:35.835237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dea5c3d2-9698-46cf-b98f-7c82eb311511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:35.899749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 448/881] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:36.163085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929de2c9-97cb-4f6b-963f-8cff30ea39f1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:36.331277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:36.374707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929de2c9-97cb-4f6b-963f-8cff30ea39f1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:36.406747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194be70c-4519-474a-8200-e5ee08dae6f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:36.407337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194be70c-4519-474a-8200-e5ee08dae6f1 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:36.407609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194be70c-4519-474a-8200-e5ee08dae6f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:36.407870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194be70c-4519-474a-8200-e5ee08dae6f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:36.408535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194be70c-4519-474a-8200-e5ee08dae6f1 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:36.409906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194be70c-4519-474a-8200-e5ee08dae6f1 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:36.425183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 434/882] 10.222.0.23 () {60 vars in 1138 bytes} [Wed Oct 12 02:18:36 2022] POST /identity/v3/domains => generated 267 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:38.560027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5795bfcd-1faa-4668-8b2e-f62d23967709 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:38.560774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5795bfcd-1faa-4668-8b2e-f62d23967709 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:38.560774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5795bfcd-1faa-4668-8b2e-f62d23967709 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:38.561928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 449/883] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:38.567247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:38.567645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:38.567786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:38.585313 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:38.586490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:38.650193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 435/884] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:38.919238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5795bfcd-1faa-4668-8b2e-f62d23967709 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:39.017917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5795bfcd-1faa-4668-8b2e-f62d23967709 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:39.025973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:39.051735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa5e47-d962-4f02-86d5-0bbf95bab1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:39.052350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa5e47-d962-4f02-86d5-0bbf95bab1ad None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:39.052625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa5e47-d962-4f02-86d5-0bbf95bab1ad None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:39.052924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa5e47-d962-4f02-86d5-0bbf95bab1ad None admin] PATH_INFO: `/v3/domains/36f794da904d4db2b824f46ce6a00a67` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:39.062860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa5e47-d962-4f02-86d5-0bbf95bab1ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=36f794da904d4db2b824f46ce6a00a67)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:39.064305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa5e47-d962-4f02-86d5-0bbf95bab1ad None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:39.065893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 450/885] 10.222.0.23 () {56 vars in 1183 bytes} [Wed Oct 12 02:18:38 2022] GET /identity/v3/domains/36f794da904d4db2b824f46ce6a00a67 => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:39.072317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:39.116950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:39.168156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34989be-4d39-4aa0-a87c-8ae53a35bc0d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:39.200070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571827d0-befc-4c5e-ad70-e7b6485384f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:39.200662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571827d0-befc-4c5e-ad70-e7b6485384f5 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:39.200960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571827d0-befc-4c5e-ad70-e7b6485384f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:39.201226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571827d0-befc-4c5e-ad70-e7b6485384f5 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:39.201876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571827d0-befc-4c5e-ad70-e7b6485384f5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:39.203284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571827d0-befc-4c5e-ad70-e7b6485384f5 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:39.228740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 436/886] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:18:39 2022] POST /identity/v3/projects => generated 361 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:41.367023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df1fa9-6169-4738-b115-ebb759b18065 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:41.367616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df1fa9-6169-4738-b115-ebb759b18065 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:41.367616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df1fa9-6169-4738-b115-ebb759b18065 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:41.369525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 451/887] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:41.374991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:41.375406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:41.375513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:41.392958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:41.394106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:41.456969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 437/888] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:41.722010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df1fa9-6169-4738-b115-ebb759b18065 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:41.821880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df1fa9-6169-4738-b115-ebb759b18065 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:41.853927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dcaa10-3fcc-417f-ba4e-6612059e41d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:41.854543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dcaa10-3fcc-417f-ba4e-6612059e41d0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:41.854819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dcaa10-3fcc-417f-ba4e-6612059e41d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:41.855088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dcaa10-3fcc-417f-ba4e-6612059e41d0 None admin] PATH_INFO: `/v3/domains/36f794da904d4db2b824f46ce6a00a67` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:41.856196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dcaa10-3fcc-417f-ba4e-6612059e41d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=36f794da904d4db2b824f46ce6a00a67)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:41.857664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dcaa10-3fcc-417f-ba4e-6612059e41d0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:41.859211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 452/889] 10.222.0.23 () {56 vars in 1183 bytes} [Wed Oct 12 02:18:41 2022] GET /identity/v3/domains/36f794da904d4db2b824f46ce6a00a67 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:41.864884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:41.892060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:41.959142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0226784e-eef8-4c4f-be42-03e2095afc1b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:41.990886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231ca187-9cff-4ce2-8c2a-82bf6f1995ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:41.991511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231ca187-9cff-4ce2-8c2a-82bf6f1995ac None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:41.991792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231ca187-9cff-4ce2-8c2a-82bf6f1995ac None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:41.992058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231ca187-9cff-4ce2-8c2a-82bf6f1995ac None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:41.992720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231ca187-9cff-4ce2-8c2a-82bf6f1995ac None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:41.994647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231ca187-9cff-4ce2-8c2a-82bf6f1995ac None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:42.015639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 438/890] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:18:41 2022] POST /identity/v3/users => generated 311 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:18:44.157321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c29647-0f48-4dda-bce1-06e21955e3c1 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:44.157766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c29647-0f48-4dda-bce1-06e21955e3c1 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:44.157865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c29647-0f48-4dda-bce1-06e21955e3c1 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:44.159207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 453/891] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:44.164263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:44.165231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:44.165374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:44.182839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:44.184009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:44.248082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 439/892] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:44.515463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c29647-0f48-4dda-bce1-06e21955e3c1 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:44.550022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:44.619300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c29647-0f48-4dda-bce1-06e21955e3c1 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:44.651479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:44.652081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:44.652351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:44.652612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:44.662633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:44.667896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:44.669089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:18:44.670052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 454/893] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:18:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:44.675105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:44.689411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:44.770276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473a348b-151c-47ef-a36d-a941ad3215bd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:44.802677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:44.803267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:44.803539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:44.803804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:44.804446 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:44.804762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:44.806144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:44.813757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 440/894] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:18:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:44.819420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:44.909769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:44.917355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07992e90-d44f-4012-a70f-0fbbebd14309 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:44.949344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:44.949947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:44.950221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:44.950488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] PATH_INFO: `/v3/users/90fc5adb2c4b4e32976a479b3d2d599e` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:44.963405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] RBAC: Authorizing `identity:get_user(user_id=90fc5adb2c4b4e32976a479b3d2d599e)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:44.964879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:44.966440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 455/895] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:44 2022] GET /identity/v3/users/90fc5adb2c4b4e32976a479b3d2d599e => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:44.972161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:45.046585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:45.068418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0358694-f8c8-41d2-92dc-23095f806a2e None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:45.100709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d41e186-a32e-4547-a47d-ce6670587b09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:45.101352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d41e186-a32e-4547-a47d-ce6670587b09 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:45.101621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d41e186-a32e-4547-a47d-ce6670587b09 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:45.101882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d41e186-a32e-4547-a47d-ce6670587b09 None admin] PATH_INFO: `/v3/projects/951ed9f3b48c48e9a7cac9e0b0a942a2` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:45.103027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d41e186-a32e-4547-a47d-ce6670587b09 None admin] RBAC: Authorizing `identity:get_project(project_id=951ed9f3b48c48e9a7cac9e0b0a942a2)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:45.104430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d41e186-a32e-4547-a47d-ce6670587b09 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:45.106054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 441/896] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:44 2022] GET /identity/v3/projects/951ed9f3b48c48e9a7cac9e0b0a942a2 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:45.111945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:45.207307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2daeb7-d013-429e-a0a2-f0c078f5fa5f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:45.239636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11aa18a-dd4a-4810-b480-805280dc38ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:45.240268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11aa18a-dd4a-4810-b480-805280dc38ec None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:45.240539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11aa18a-dd4a-4810-b480-805280dc38ec None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:45.240834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11aa18a-dd4a-4810-b480-805280dc38ec None admin] PATH_INFO: `/v3/projects/951ed9f3b48c48e9a7cac9e0b0a942a2/users/90fc5adb2c4b4e32976a479b3d2d599e/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:45.242687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11aa18a-dd4a-4810-b480-805280dc38ec None admin] RBAC: Authorizing `identity:create_grant(project_id=951ed9f3b48c48e9a7cac9e0b0a942a2, user_id=90fc5adb2c4b4e32976a479b3d2d599e, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:45.244102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11aa18a-dd4a-4810-b480-805280dc38ec None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:45.256383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 456/897] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:18:45 2022] PUT /identity/v3/projects/951ed9f3b48c48e9a7cac9e0b0a942a2/users/90fc5adb2c4b4e32976a479b3d2d599e/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:18:47.347001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3c7f49-56c8-45a0-b98b-dcf1c39963a1 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:47.347544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3c7f49-56c8-45a0-b98b-dcf1c39963a1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:47.347544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3c7f49-56c8-45a0-b98b-dcf1c39963a1 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:47.348866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 442/898] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:18:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:18:47.354633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:47.355063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:47.355162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:47.373849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:18:47.374960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:18:47.382288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:47.441922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 457/899] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:18:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:18:47.704706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3c7f49-56c8-45a0-b98b-dcf1c39963a1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:47.801480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3c7f49-56c8-45a0-b98b-dcf1c39963a1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:47.832193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:47.832816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:47.833092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:47.833352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:47.843797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:18:47.848483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:18:47.849609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:18:47.850649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 443/900] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:18:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:18:47.855884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:47.950874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e51d4-d0b3-4719-9f04-c4a46e70c19d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:47.984336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:47.984894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:47.985168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:47.985433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:47.986080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:47.986376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:47.987742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:47.995509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 458/901] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:18:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:48.001628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:48.005349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:48.096988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea2991-d1b6-4763-99b0-6df38e7eb189 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:48.130919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:48.131543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:48.131818 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:48.132084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] PATH_INFO: `/v3/users/90fc5adb2c4b4e32976a479b3d2d599e` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:48.133738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] RBAC: Authorizing `identity:get_user(user_id=90fc5adb2c4b4e32976a479b3d2d599e)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:48.135150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:48.136743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 444/902] 10.222.0.23 () {56 vars in 1177 bytes} [Wed Oct 12 02:18:48 2022] GET /identity/v3/users/90fc5adb2c4b4e32976a479b3d2d599e => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:48.142651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:48.146420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:48.239283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011bd6c1-2ad6-460b-9f8a-6e65b7f279a4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:48.271580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:48.272163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:48.272437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:48.272729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:48.337496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:48.338932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:48.341425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:48.433813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 459/903] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:18:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:18:48.439628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:48.509612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:48.538266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fe49a9-941d-46ce-97e1-915ae0b8711d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:48.570111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436bf2a3-64f6-4de9-b6ac-87ff5afd73aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:48.571250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436bf2a3-64f6-4de9-b6ac-87ff5afd73aa None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:48.571524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436bf2a3-64f6-4de9-b6ac-87ff5afd73aa None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:48.571788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436bf2a3-64f6-4de9-b6ac-87ff5afd73aa None admin] PATH_INFO: `/v3/projects/951ed9f3b48c48e9a7cac9e0b0a942a2` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:48.572912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436bf2a3-64f6-4de9-b6ac-87ff5afd73aa None admin] RBAC: Authorizing `identity:get_project(project_id=951ed9f3b48c48e9a7cac9e0b0a942a2)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:48.574329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436bf2a3-64f6-4de9-b6ac-87ff5afd73aa None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:48.575924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 445/904] 10.222.0.23 () {56 vars in 1186 bytes} [Wed Oct 12 02:18:48 2022] GET /identity/v3/projects/951ed9f3b48c48e9a7cac9e0b0a942a2 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:48.581933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:18:48.677091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb96576-fd49-4d16-8797-ae5a83e09edb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:18:48.679328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:48.710952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:18:48.711542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:18:48.711819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:18:48.712085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:18:48.712823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=951ed9f3b48c48e9a7cac9e0b0a942a2, user.id=90fc5adb2c4b4e32976a479b3d2d599e) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:18:48.713130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:18:48.714553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a298f-d6cd-4d5e-a62a-009dde3bdf7b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:18:48.720140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 460/905] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:18:48 2022] GET /identity/v3/role_assignments?user.id=90fc5adb2c4b4e32976a479b3d2d599e&scope.project.id=951ed9f3b48c48e9a7cac9e0b0a942a2&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:18:58.566091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=30 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:18:58.717592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=16 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:43.765615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2b528-a4d5-45b7-be4b-ab93b23d488e None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:43.766291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2b528-a4d5-45b7-be4b-ab93b23d488e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:43.766291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2b528-a4d5-45b7-be4b-ab93b23d488e None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:43.767833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 446/906] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:19:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:19:43.775388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:43.775944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:43.775944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:43.797595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:19:43.798833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:19:43.879637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 461/907] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:19:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:19:44.154945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2b528-a4d5-45b7-be4b-ab93b23d488e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:44.284884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2b528-a4d5-45b7-be4b-ab93b23d488e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:44.330844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:44.331886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:44.332358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:44.332997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:44.345583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:44.347322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:44.366750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:44.368004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:44.369007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:44.369007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:44.369007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:44.369007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 447/908] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:44.374542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:44.492557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb291e2-fe49-4d63-a529-b9110ee1edef None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:44.526783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:44.533779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:44.534603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:44.534971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:44.535308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:44.536576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:44.537003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:44.538586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c8d94-3be5-461e-80d6-42ab1bd9008b None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:44.553585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 462/909] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:19:44.559279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:44.630669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:44.696536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ac018-33d8-4899-8eca-5dc6936727f8 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:44.735070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a62f673-6948-47eb-8524-3e5aeac8cb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:44.735826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62f673-6948-47eb-8524-3e5aeac8cb25 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:44.736144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62f673-6948-47eb-8524-3e5aeac8cb25 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:44.736419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62f673-6948-47eb-8524-3e5aeac8cb25 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:44.737201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a62f673-6948-47eb-8524-3e5aeac8cb25 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:44.738747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a62f673-6948-47eb-8524-3e5aeac8cb25 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:44.766393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 448/910] 10.222.0.23 () {60 vars in 1132 bytes} [Wed Oct 12 02:19:44 2022] POST /identity/v3/users => generated 251 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:19:47.362651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ba2b6d-ae6f-40cd-a373-b78ea6b53ca6 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:47.363714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ba2b6d-ae6f-40cd-a373-b78ea6b53ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:47.363714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ba2b6d-ae6f-40cd-a373-b78ea6b53ca6 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:47.364606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 463/911] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:19:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:19:47.370341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:47.370762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:47.370872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:47.391652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:19:47.392882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:19:47.465488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 449/912] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:19:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:19:47.765319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ba2b6d-ae6f-40cd-a373-b78ea6b53ca6 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:47.882126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ba2b6d-ae6f-40cd-a373-b78ea6b53ca6 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:47.920982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:47.921660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:47.921955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:47.922289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:47.935558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:19:47.940797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:47.941949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:47.943517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 464/913] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:19:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:19:47.955015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:48.002196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:48.069547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11381a72-583e-41b8-aecb-187d79a0c066 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:48.104846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:48.105508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:48.105780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:48.106047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:48.106878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] RBAC: Adding query filter params (name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:48.107182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:48.108977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:48.123423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 450/914] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:19:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:48.124666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:48.182432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:48.240639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15aad9d8-ce91-402b-a2ea-6fb10d0980f1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:48.279963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:48.280595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:48.280898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:48.281179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:48.293322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:48.295026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:48.311404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:48.312579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:48.313514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:48.313514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:48.313514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:48.313514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 465/915] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:48.319714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:48.428961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:48.432209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebaabc-e47d-485d-9697-6cb4eb0c6c23 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:48.470414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:48.471024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:48.471298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:48.471565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:48.472196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:48.472491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:48.474436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:48.486475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 451/916] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:48.492485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:48.607505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaba0f4-d86b-4369-b160-8f1f9b7d066e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:48.609241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:48.650176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:48.650824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:48.651098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:48.651364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:48.661589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:48.663133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:19:48.682672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:48.688552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:48.689780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 466/917] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:48.690537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:48.811928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2913022a-08cf-427b-b9a1-31fca9a65f57 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:48.850952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:48.851563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:48.851860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:48.852103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:48.852852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:48.853188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:48.854750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:48.872689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 452/918] 10.222.0.23 () {56 vars in 1143 bytes} [Wed Oct 12 02:19:48 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:19:48.878946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:48.989689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ff586-43df-471e-8bab-d82f16a87725 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:49.028660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:49.029315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:49.029588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:49.029855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:49.041298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:49.042807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:49.061079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:49.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:49.063202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:49.063202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:49.063202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:49.063202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 467/919] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:49.068479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:49.091365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:49.179423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f90b15-fa50-4c5d-8a79-b1e80d9de8d9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:49.221225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:49.221225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:49.221225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:49.221225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:49.221700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:49.221700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:49.221864 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:49.237294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 453/920] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:49.243145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:49.263485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:49.381657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2eb6886-672d-41c8-ba0d-06d9a64cac82 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:49.426224 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:49.427321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:49.427608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:49.427893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:49.435622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:49.437050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:19:49.451789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:49.453068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:49.453952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 468/921] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:19:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:49.459687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:49.568900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:49.594879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82b2b56-aeee-4ab9-a130-6b30efe50040 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:49.642115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:49.642766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:49.643089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:49.643517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:49.644185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:49.644485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:49.646633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2769d-d8ad-4fdc-91c5-9613d6e87141 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:49.661043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 454/922] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:19:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:19:49.669117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:49.766015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:49.796997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ae49ad-dfa9-4af8-9473-700f91aec19f None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:49.836128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afdd43-1f5b-4ba8-98a4-ceb622bfcf9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:49.836963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afdd43-1f5b-4ba8-98a4-ceb622bfcf9e None admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:49.837239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afdd43-1f5b-4ba8-98a4-ceb622bfcf9e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:49.837507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afdd43-1f5b-4ba8-98a4-ceb622bfcf9e None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/38f000e12e7b4cd1b34875f6c0bf9cc4/roles/87b12fe9315f448487f939cad9054a60` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:49.855422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afdd43-1f5b-4ba8-98a4-ceb622bfcf9e None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=38f000e12e7b4cd1b34875f6c0bf9cc4, role_id=87b12fe9315f448487f939cad9054a60)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:49.857313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afdd43-1f5b-4ba8-98a4-ceb622bfcf9e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:49.873388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 469/923] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:19:49 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/38f000e12e7b4cd1b34875f6c0bf9cc4/roles/87b12fe9315f448487f939cad9054a60 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:19:52.095782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e75668-dcf7-4c30-ac19-ba2e80eae5e0 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:52.096320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e75668-dcf7-4c30-ac19-ba2e80eae5e0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:52.096400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e75668-dcf7-4c30-ac19-ba2e80eae5e0 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:52.098200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 455/924] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:19:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:19:52.105289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:52.105747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:52.105833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:52.124984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:19:52.126167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:19:52.197691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 470/925] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:19:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:19:52.462165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e75668-dcf7-4c30-ac19-ba2e80eae5e0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:52.571684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e75668-dcf7-4c30-ac19-ba2e80eae5e0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:52.613471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:52.614656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:52.614935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:52.615205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:52.622873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:52.627267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:19:52.633623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:52.634772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:52.635602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 456/926] 10.222.0.23 () {56 vars in 1102 bytes} [Wed Oct 12 02:19:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:52.641452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:52.699694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:52.750728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c76bc5a-9663-45a0-b260-949a9d034b1f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:52.786848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:52.787509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:52.787821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:52.788092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:52.788771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:52.789071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:52.790454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:52.799117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 471/927] 10.222.0.23 () {56 vars in 1103 bytes} [Wed Oct 12 02:19:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:52.805784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:52.914596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22766338-98b8-4c56-8337-5185b4e04e5c None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:52.951604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:52.952296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:52.952563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:52.952857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:52.963979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:52.965430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:52.980273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:52.981393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:52.982506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:52.982506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:52.982506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:52.982506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 457/928] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:19:52.987846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:53.094322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b055b626-bdcd-4b0b-813c-3193d99c6fc0 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:53.111333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:53.130782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:53.131386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:53.131657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:53.131924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:53.132546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:53.132870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:53.134233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:53.146836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 472/929] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:53.152479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:53.208938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:53.264133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c872dd0a-7b3a-462c-bc72-f7db48b81b0d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:53.305161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:53.305918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:53.306234 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:53.306568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:53.314535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:53.315966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:19:53.327270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:53.328844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:53.330054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 458/930] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:53.334897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:53.438119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca2dae0-cffe-4a29-834d-a674515ba4ba None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:53.475426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:53.476069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:53.476344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:53.476610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:53.477369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:53.477685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:53.479109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:53.497026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 473/931] 10.222.0.23 () {56 vars in 1143 bytes} [Wed Oct 12 02:19:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:53.504873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:53.612590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9f9f3a-ef05-437b-bc02-c31b3fca64b2 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:53.624483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:53.650064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:53.650685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:53.650992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:53.651231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:53.729741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:53.731302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:53.843624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 459/932] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:19:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:19:53.849635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:53.887290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:53.960266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9f59b-3302-42fb-8ece-89cfdefc03b9 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:53.999605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:54.000153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:54.000467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:54.000792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:54.012689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:54.014400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:54.029396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:54.030657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:54.031720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:54.031720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:54.031720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:54.031720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 474/933] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:54.037064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:54.050402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:54.155182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1839e42-b0c8-4d8f-b75a-175d9efeddc9 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:54.192909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:54.193521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:54.193796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:54.194063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:54.194697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:54.195007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:54.196600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:54.208633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 460/934] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:54.214071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:54.305493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:54.320674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c363ee5c-6a3c-40f3-ada3-e31ff8dad6eb None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:54.357414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:54.358031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:54.358301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:54.358574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:54.365640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:54.367077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:19:54.378538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:54.379814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:19:54.380969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:19:54.384414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 475/935] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:19:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:54.385672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:54.463726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:54.499411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48718229-a4d0-4be0-ba21-f9ecda1e22a8 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:54.536842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:54.537510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:54.537920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:54.538244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:54.539049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:54.539347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:54.540770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aef0b13-a62a-4187-a007-5d161619842d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:54.555282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 461/936] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:19:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:54.561936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:54.672789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c88ac3b-4df3-469a-97eb-dc5a84fd557e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:54.707503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:54.708101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:54.708372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:54.708635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:54.709370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] RBAC: Adding query filter params (role.id=87b12fe9315f448487f939cad9054a60, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=38f000e12e7b4cd1b34875f6c0bf9cc4) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:54.709668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:54.711086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3cc0e1-b498-40b1-86f7-e1a92176820a None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:54.716717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 476/937] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:19:54 2022] GET /identity/v3/role_assignments?user.id=38f000e12e7b4cd1b34875f6c0bf9cc4&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=87b12fe9315f448487f939cad9054a60 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:57.043270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0cb5e9-77ea-485c-b260-4fe41f53544d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:57.043795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0cb5e9-77ea-485c-b260-4fe41f53544d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:57.043795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0cb5e9-77ea-485c-b260-4fe41f53544d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:57.045219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 462/938] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:19:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:19:57.050574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:57.051639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:57.051639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:57.069667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:19:57.070839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:19:57.073668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:57.145503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 477/939] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:19:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 12 02:19:57.413820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0cb5e9-77ea-485c-b260-4fe41f53544d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:57.521974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0cb5e9-77ea-485c-b260-4fe41f53544d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:57.553801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:57.559965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:57.560671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:57.561119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:57.561389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:57.572786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:19:57.578368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:57.579477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:57.580310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 463/940] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:19:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:57.585777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:57.696905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15879cb1-e13d-4252-8447-09c305ff5532 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:57.738410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:57.739043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:57.739316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:57.739627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:57.740288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:57.740586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:57.742077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:57.751633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 478/941] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:19:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:57.758714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:57.872953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef7c13-0dd6-4404-a25a-441ee3f7c912 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:57.911487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:57.912103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:57.912362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:57.912633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:57.926855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:57.929061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:57.953933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:57.955036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:57.956035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:57.956035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:57.956035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:57.956035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 464/942] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:19:57.962227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:57.965240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:58.069562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45268b7d-2c4c-4dd9-a870-d1ea5a04077e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:58.111721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:58.112401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:58.112762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:58.113041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:58.114404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:58.115053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:58.117048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:58.133195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 479/943] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:58.139502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:58.198870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:58.263373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c154db-d043-49d2-950d-d20c30af9711 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:58.303679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:58.304308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:58.304581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:58.304886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] PATH_INFO: `/v3/users/placement` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:58.314528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:58.316000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:19:58.329700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:58.330728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:58.331665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 465/944] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:58.337518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:58.408642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:58.454710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b52a1-3c1a-4a6b-b836-49ebf7865ca4 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:58.492408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:58.493049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:58.493320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:58.493585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:58.494293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:58.494587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:58.496494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:58.516016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 480/945] 10.222.0.23 () {56 vars in 1143 bytes} [Wed Oct 12 02:19:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:58.521953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:58.637812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58b1509-dbd0-4a20-8e8d-d2c02d44d6ff None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:58.639377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:58.678357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:58.679029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:58.679287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:58.679597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:58.692188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:58.693811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:19:58.709171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:58.710332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:19:58.711521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:19:58.711521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:19:58.711521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:58.711521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 466/946] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:19:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:19:58.716466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:58.836865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa036d8-64bb-4918-8584-4afbe3c43c75 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:58.859724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:58.875009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:58.875621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:58.875957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:58.876602 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:58.877295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:58.877587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:58.879004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:58.890628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 481/947] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:19:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:58.896966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:59.019657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f078002e-d967-4d52-8be2-89175eb32265 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:59.058055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:59.058762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:59.059049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:59.059333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:59.066431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:59.067878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:19:59.081087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:19:59.081971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:19:59.083153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 467/948] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:19:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 12 02:19:59.088947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:59.208485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d87bbc-6d21-4b41-b38c-242d356485d1 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:59.248442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:59.249092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:59.249370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:59.249667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:59.250350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:19:59.250653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:59.252086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c355b-97d9-4049-a8e3-38f57b0d8e29 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:59.265398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 482/949] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:19:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:19:59.273195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:19:59.291777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:19:59.390007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81bbf9a-544f-40f7-b34e-56ae36b8c579 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:19:59.427861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfef923-3d66-425c-9d7a-400581956044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:19:59.428640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfef923-3d66-425c-9d7a-400581956044 None admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:19:59.428944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfef923-3d66-425c-9d7a-400581956044 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:19:59.429217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfef923-3d66-425c-9d7a-400581956044 None admin] PATH_INFO: `/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/38f000e12e7b4cd1b34875f6c0bf9cc4/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:19:59.431596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfef923-3d66-425c-9d7a-400581956044 None admin] RBAC: Authorizing `identity:create_grant(project_id=0e6c7a0acc5d40ffb16cb54d7c51720f, user_id=38f000e12e7b4cd1b34875f6c0bf9cc4, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:19:59.433119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfef923-3d66-425c-9d7a-400581956044 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:19:59.448491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 468/950] 10.222.0.23 () {58 vars in 1439 bytes} [Wed Oct 12 02:19:59 2022] PUT /identity/v3/projects/0e6c7a0acc5d40ffb16cb54d7c51720f/users/38f000e12e7b4cd1b34875f6c0bf9cc4/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:20:01.847260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d9a9c-0f04-42dd-baab-2a9a05b1928f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:01.848324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d9a9c-0f04-42dd-baab-2a9a05b1928f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:01.848324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d9a9c-0f04-42dd-baab-2a9a05b1928f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:01.849667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 483/951] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:20:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:20:01.855253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:01.855795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:01.855795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:01.875139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:20:01.876325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:20:01.948581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 469/952] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:20:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:20:02.258906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d9a9c-0f04-42dd-baab-2a9a05b1928f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:02.312428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:02.378926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d9a9c-0f04-42dd-baab-2a9a05b1928f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:02.418685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:02.419336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:02.419612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:02.419879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:02.432120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 12 02:20:02.437054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:02.438130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:02.438991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 484/953] 10.222.0.23 () {56 vars in 1096 bytes} [Wed Oct 12 02:20:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:02.444534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:02.447746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:02.547497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae86f39-bb45-4f91-b714-12d50112a0ee None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:02.581975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:02.582590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:02.582861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:02.583124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:02.583780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:02.584074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:02.585476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:02.594498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 470/954] 10.222.0.23 () {56 vars in 1099 bytes} [Wed Oct 12 02:20:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:02.600857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:02.703840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:02.707900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d781279-5f8d-4653-87c8-8eea9e1b3487 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:02.749517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:02.750711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:02.751075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:02.751656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:02.771916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:02.774357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:20:02.792637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:02.793657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:20:02.794574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:20:02.794574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:20:02.794574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:02.794574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 485/955] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:20:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:02.800912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:02.870591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:02.907534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcb1705-c9ae-48e5-b0ce-037806c3248d None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:02.940398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:02.941031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:02.941328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:02.941593 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:02.942220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:02.942515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:02.943858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:02.955607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 471/956] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:20:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:02.961730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:03.064720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47875b0-f388-4daa-8ce5-34bb3cf07277 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:03.096198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:03.096850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:03.097119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:03.097938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:03.106053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:03.107499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 12 02:20:03.118471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:03.119574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:03.121126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 486/957] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:20:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:03.125649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:03.219888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f2eee-f336-42dc-9f79-466c621b32b6 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:03.243468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:03.251985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:03.252577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:03.252883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:03.253141 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:03.254357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:03.254652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:03.256039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:03.272774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 472/958] 10.222.0.23 () {56 vars in 1143 bytes} [Wed Oct 12 02:20:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:03.278363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:03.293171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:03.375706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ec0f7-045f-48d7-8e53-7432c46afb98 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:03.407763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:03.408345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:03.408616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:03.408906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:03.472496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:03.473899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:03.486455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:03.568711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 487/959] 10.222.0.23 () {58 vars in 1282 bytes} [Wed Oct 12 02:20:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 12 02:20:03.574351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:03.668187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631bc5d2-b38c-4981-9a0c-1071e8587175 None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:03.700616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:03.701245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:03.701516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:03.701780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:03.711256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:03.712663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 12 02:20:03.728174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:03.729189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 12 02:20:03.730092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 12 02:20:03.730092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 12 02:20:03.730092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:20:03.730092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 473/960] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:20:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:03.735124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:03.835209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fb3f4e-e95f-4e41-aecd-99cbad337eb3 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:03.847600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:03.867877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:03.868462 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:03.868755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:03.869020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:03.869895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:03.870185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:03.871545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:03.882799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 488/961] 10.222.0.23 () {56 vars in 1109 bytes} [Wed Oct 12 02:20:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:03.888563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:03.922998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:03.985037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a51712-9a14-40cd-a1cd-b42b57e5de5f None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:04.017215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:04.017823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:04.018096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:04.018359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:04.024764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:04.026191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 12 02:20:04.037547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:04.038582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: ERROR keystone.server.flask.application Oct 12 02:20:04.039693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 474/962] 10.222.0.23 () {56 vars in 1111 bytes} [Wed Oct 12 02:20:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:04.044386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:04.143717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976c6a37-9485-4e53-80e0-291155fabf4e None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:04.176172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:04.176787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:04.177060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:04.177322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:04.177987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:04.178284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:04.179695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118aed5-185e-4f9c-8b50-ddad4ea0f77f None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:04.191067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 489/963] 10.222.0.23 () {56 vars in 1148 bytes} [Wed Oct 12 02:20:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:04.197052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:04.289192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:04.295048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81372c7-1f56-4736-8852-42af516387bf None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:04.326638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:04.327222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:04.327493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:04.327756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:04.328458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] RBAC: Adding query filter params (role.id=32a3720c5c9442649cd9a8c93b3bbbd7, scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f, user.id=38f000e12e7b4cd1b34875f6c0bf9cc4) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:04.328784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:04.330191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c6073-66be-4033-b934-e21f0cce4d12 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:04.336197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 475/964] 10.222.0.23 () {56 vars in 1374 bytes} [Wed Oct 12 02:20:04 2022] GET /identity/v3/role_assignments?user.id=38f000e12e7b4cd1b34875f6c0bf9cc4&scope.project.id=0e6c7a0acc5d40ffb16cb54d7c51720f&role.id=32a3720c5c9442649cd9a8c93b3bbbd7 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:06.469432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60235423-ee2a-4f7a-9f53-26279da66ca1 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:06.469952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60235423-ee2a-4f7a-9f53-26279da66ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:06.469952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60235423-ee2a-4f7a-9f53-26279da66ca1 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:06.471314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 490/965] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:20:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:20:06.476445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:06.476894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:06.477076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:06.495353 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:20:06.496515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:20:06.559481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 476/966] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:20:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:20:06.845873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60235423-ee2a-4f7a-9f53-26279da66ca1 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:06.876252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:06.946096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60235423-ee2a-4f7a-9f53-26279da66ca1 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:06.978814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:06.979425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:06.979695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:06.979957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] PATH_INFO: `/v3/services/placement` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:06.980527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:06.981926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:06.990500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:06.991816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:06.992984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 491/967] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:20:06 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:06.997768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:07.084951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:07.094473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dece46d-40f1-4867-84da-267d3e32e131 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:07.127808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:07.129043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:07.129317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:07.129579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:07.130198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] RBAC: Adding query filter params (name=placement) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:07.130493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:07.131837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c18361-7383-42f8-9bfc-b70f4124761b None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:07.136444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 477/968] 10.222.0.23 () {56 vars in 1116 bytes} [Wed Oct 12 02:20:06 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:07.142104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:07.232879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:07.239461 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8bec4-6093-419f-9745-468c342c9fca None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:07.271993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:07.272580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:07.272882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:07.273149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] PATH_INFO: `/v3/services` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:07.273758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:07.274052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:07.275414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d08197e-753a-4b4d-902b-f12d6b39b741 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:07.279664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 492/969] 10.222.0.23 () {56 vars in 1116 bytes} [Wed Oct 12 02:20:07 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:09.376066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd49da0c-b860-47ff-99a0-05cfd80b8f1a None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:09.376571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd49da0c-b860-47ff-99a0-05cfd80b8f1a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:09.376571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd49da0c-b860-47ff-99a0-05cfd80b8f1a None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:09.377939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 478/970] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:20:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:20:09.383018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ebf73-1bcd-464b-96c4-d2bc70c8f04d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:09.383378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ebf73-1bcd-464b-96c4-d2bc70c8f04d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:09.383560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ebf73-1bcd-464b-96c4-d2bc70c8f04d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:09.401830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-2a3ebf73-1bcd-464b-96c4-d2bc70c8f04d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:20:09.403008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2a3ebf73-1bcd-464b-96c4-d2bc70c8f04d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:20:09.467194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 493/971] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:20:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:20:09.734016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd49da0c-b860-47ff-99a0-05cfd80b8f1a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:09.837469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd49da0c-b860-47ff-99a0-05cfd80b8f1a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:09.870340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5260da-bebe-4df6-8c95-c9b1da062ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:09.870945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5260da-bebe-4df6-8c95-c9b1da062ed3 None admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:09.871221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5260da-bebe-4df6-8c95-c9b1da062ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:09.871491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5260da-bebe-4df6-8c95-c9b1da062ed3 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:09.872148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5260da-bebe-4df6-8c95-c9b1da062ed3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:09.873525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5260da-bebe-4df6-8c95-c9b1da062ed3 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:09.886936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 479/972] 10.222.0.23 () {60 vars in 1141 bytes} [Wed Oct 12 02:20:09 2022] POST /identity/v3/services => generated 246 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:20:12.024635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81358a87-0ffe-4576-8bb6-8e617772501f None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:12.025258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81358a87-0ffe-4576-8bb6-8e617772501f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:12.025258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81358a87-0ffe-4576-8bb6-8e617772501f None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:12.026553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 494/973] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:20:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:20:12.031820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:12.032209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:12.032356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:12.051422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:20:12.052611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:20:12.061559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:12.125127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 480/974] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:20:12 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:20:12.389455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81358a87-0ffe-4576-8bb6-8e617772501f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:12.478236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:12.488915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81358a87-0ffe-4576-8bb6-8e617772501f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:12.522215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:12.522826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:12.523100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:12.523367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:12.523937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:12.525414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:12.533993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:12.535785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:12.536931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 495/975] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:20:12 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:12.541501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:12.634068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf3ffb-fb40-4f57-bc5b-9dedf939fde3 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:12.666522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:12.667056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:12.667326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:12.667586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:12.668195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] RBAC: Adding query filter params (name=placement) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:12.668488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:12.669856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:12.674706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 481/976] 10.222.0.23 () {56 vars in 1116 bytes} [Wed Oct 12 02:20:12 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:12.680464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:12.792586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd754dd-8f21-4973-9c6e-2eef0db1c498 None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:12.827047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:12.827645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:12.827920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:12.828188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:12.828906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] RBAC: Adding query filter params (interface=public, service_id=99adf3bc463f4747b08f07897d04ce44, region_id=RegionOne) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:12.829213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:12.830586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7024b8e6-f06a-4a31-ac07-c0e8476824c4 None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:12.835958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 496/977] 10.222.0.23 () {56 vars in 1251 bytes} [Wed Oct 12 02:20:12 2022] GET /identity/v3/endpoints?service_id=99adf3bc463f4747b08f07897d04ce44&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:12.841741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:12.845414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:12.939623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9d6621-4e0a-4d5b-a683-45d0115748ae None admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:12.972002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b99ec2-f931-4a46-8399-41ff6d950d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:12.972587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b99ec2-f931-4a46-8399-41ff6d950d59 None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:12.972905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b99ec2-f931-4a46-8399-41ff6d950d59 None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:12.973172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b99ec2-f931-4a46-8399-41ff6d950d59 None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:12.973754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b99ec2-f931-4a46-8399-41ff6d950d59 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:12.975100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b99ec2-f931-4a46-8399-41ff6d950d59 None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:12.981662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 482/978] 10.222.0.23 () {56 vars in 1087 bytes} [Wed Oct 12 02:20:12 2022] GET /identity/v3/services => generated 2191 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:20:15.094301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd100a4a-277b-40eb-8cfc-ca6842d5d0de None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:15.094857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd100a4a-277b-40eb-8cfc-ca6842d5d0de None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:15.094857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd100a4a-277b-40eb-8cfc-ca6842d5d0de None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:15.096161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 497/979] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:20:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:20:15.101343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:15.101785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:15.101894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:15.119739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:20:15.120941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:20:15.179571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:15.185298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 483/980] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:20:15 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 12 02:20:15.451501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd100a4a-277b-40eb-8cfc-ca6842d5d0de None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:15.471011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:15.551496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd100a4a-277b-40eb-8cfc-ca6842d5d0de None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:15.583749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:15.584356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:15.584628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:15.584919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] PATH_INFO: `/v3/services/placement` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:15.585490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:15.586854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 12 02:20:15.595415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 12 02:20:15.596541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.get_or_create( Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application with Lock( Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self._enter() Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application return self.creator() Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application created_value = creator( Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: ERROR keystone.server.flask.application Oct 12 02:20:15.597967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 498/981] 10.222.0.23 () {56 vars in 1117 bytes} [Wed Oct 12 02:20:15 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 12 02:20:15.602906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:15.697264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd896fc-16f0-4472-bfa7-477b3db51342 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:15.732285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:15.732923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:15.733233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:15.733600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] PATH_INFO: `/v3/services` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:15.734259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:20:15.734564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:15.736087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eaa202-615c-4905-827b-9a687cc4104f None admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:15.741902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 484/982] 10.222.0.23 () {56 vars in 1116 bytes} [Wed Oct 12 02:20:15 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:20:15.748029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:20:15.842545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:15.860609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e214ae-ca8a-4778-b9bc-b401e7d8a7db None admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:20:15.894339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea67ccb-5a75-4536-99e6-fb77c6831bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:20:15.894930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea67ccb-5a75-4536-99e6-fb77c6831bab None admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:20:15.895201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea67ccb-5a75-4536-99e6-fb77c6831bab None admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:20:15.895464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea67ccb-5a75-4536-99e6-fb77c6831bab None admin] PATH_INFO: `/v3/endpoints` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:20:15.896113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea67ccb-5a75-4536-99e6-fb77c6831bab None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:20:15.897499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea67ccb-5a75-4536-99e6-fb77c6831bab None admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:20:15.915959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 499/983] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:20:15 2022] POST /identity/v3/endpoints => generated 331 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:20:25.738750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=69 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:20:25.911762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:19.118195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1408f-e2c2-4b9b-a4ad-8a37f7d714a7 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:19.118655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1408f-e2c2-4b9b-a4ad-8a37f7d714a7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:19.118870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1408f-e2c2-4b9b-a4ad-8a37f7d714a7 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:19.120159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 485/984] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:19.125535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592454c-1800-4019-ba7c-9a04256b534f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:19.125952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592454c-1800-4019-ba7c-9a04256b534f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:19.126144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592454c-1800-4019-ba7c-9a04256b534f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:19.226108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a592454c-1800-4019-ba7c-9a04256b534f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:19.227580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a592454c-1800-4019-ba7c-9a04256b534f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:19.354396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 500/985] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:19.647575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ac872-4bb0-424c-aa34-fe6d56e040ae None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:19.647928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ac872-4bb0-424c-aa34-fe6d56e040ae None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:19.648117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ac872-4bb0-424c-aa34-fe6d56e040ae None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:19.649752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 486/986] 10.222.0.23 () {54 vars in 964 bytes} [Wed Oct 12 02:21:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:19.656124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c5a722-983e-4ad3-8b0e-571e121b92e9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:19.656408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c5a722-983e-4ad3-8b0e-571e121b92e9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:19.656684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c5a722-983e-4ad3-8b0e-571e121b92e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:19.740021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-02c5a722-983e-4ad3-8b0e-571e121b92e9 None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:19.741904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-02c5a722-983e-4ad3-8b0e-571e121b92e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:19.765375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:19.829025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 501/987] 10.222.0.23 () {58 vars in 1061 bytes} [Wed Oct 12 02:21:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:21:19.837999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ac872-4bb0-424c-aa34-fe6d56e040ae None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:19.839747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ac2ac872-4bb0-424c-aa34-fe6d56e040ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:19.978530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ac872-4bb0-424c-aa34-fe6d56e040ae None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:20.068731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:20.069355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:20.069635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:20.069899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:20.071572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:20.134376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:20.195686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:20.197159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9992c3-cfa2-4d3a-b550-7f594968eebd service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:20.216947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 487/988] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:22.728206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb3efa-f955-4040-92a0-7f43df71c08c None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:22.728817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb3efa-f955-4040-92a0-7f43df71c08c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:22.728817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb3efa-f955-4040-92a0-7f43df71c08c None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:22.730216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 502/989] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:22.735900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60ae224-dd68-48e3-8a89-a34995ebb12b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:22.736338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60ae224-dd68-48e3-8a89-a34995ebb12b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:22.736430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60ae224-dd68-48e3-8a89-a34995ebb12b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:22.817968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f60ae224-dd68-48e3-8a89-a34995ebb12b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:22.819206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f60ae224-dd68-48e3-8a89-a34995ebb12b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:22.917499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 488/990] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:23.205188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb3efa-f955-4040-92a0-7f43df71c08c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:23.206749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8ffb3efa-f955-4040-92a0-7f43df71c08c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:23.313575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb3efa-f955-4040-92a0-7f43df71c08c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:23.383550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:23.385983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:23.386802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:23.387266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:23.387724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:23.389394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:23.496253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:23.497726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3e48f-84fb-4a9d-9c7c-5d9c6c16dddd service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:23.516408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 503/991] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:25.944629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55ee0f1-f70b-4926-b469-17e9f9b489cf None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:25.945220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55ee0f1-f70b-4926-b469-17e9f9b489cf None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:25.945220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55ee0f1-f70b-4926-b469-17e9f9b489cf None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:25.946582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 489/992] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:25.952364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ca3cf-0544-432b-9642-abd53368c273 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:25.952792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ca3cf-0544-432b-9642-abd53368c273 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:25.952994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ca3cf-0544-432b-9642-abd53368c273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:26.017213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ed9ca3cf-0544-432b-9642-abd53368c273 None None] MFA Rules not processed for user `e66d8b7ddd4d4982adf0584a255beaf1`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:26.018395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ed9ca3cf-0544-432b-9642-abd53368c273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:26.068879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:26.114029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 504/993] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:25 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:26.394515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55ee0f1-f70b-4926-b469-17e9f9b489cf None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:26.396054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d55ee0f1-f70b-4926-b469-17e9f9b489cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:26.406647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:26.497690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55ee0f1-f70b-4926-b469-17e9f9b489cf None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:26.569872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:26.570468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:26.570745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:26.571012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:26.572589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:26.634525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:26.681385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:26.682744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211c8ef4-2fe3-4929-8a7c-3898cdc8dd83 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:26.700159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 490/994] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:26 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:29.340724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df358c9a-6e1e-4ca4-b824-da9418be9c1b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:29.341280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df358c9a-6e1e-4ca4-b824-da9418be9c1b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:29.341280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df358c9a-6e1e-4ca4-b824-da9418be9c1b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:29.343169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 505/995] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:29.348502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c91edd-4ec6-4d4d-bb27-848709c1628c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:29.348912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c91edd-4ec6-4d4d-bb27-848709c1628c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:29.349101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c91edd-4ec6-4d4d-bb27-848709c1628c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:29.414309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-91c91edd-4ec6-4d4d-bb27-848709c1628c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:29.415458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-91c91edd-4ec6-4d4d-bb27-848709c1628c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:29.510570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 491/996] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:29 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:29.790166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df358c9a-6e1e-4ca4-b824-da9418be9c1b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:29.791701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-df358c9a-6e1e-4ca4-b824-da9418be9c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:29.895286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df358c9a-6e1e-4ca4-b824-da9418be9c1b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:29.967443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:29.968044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:29.968320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:29.968589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:29.970188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:29.976943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:30.077411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:30.078783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10657e0f-9bd5-405c-8945-e1a8699f2b62 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:30.096957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 506/997] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:32.907904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d0dd2b-3873-41a5-894a-a8fa19e094cf None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:32.908468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d0dd2b-3873-41a5-894a-a8fa19e094cf None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:32.908468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d0dd2b-3873-41a5-894a-a8fa19e094cf None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:32.909850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 492/998] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:32.915801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d87ad0-ff81-4c19-889b-340a2013c07a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:32.916349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d87ad0-ff81-4c19-889b-340a2013c07a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:32.916349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d87ad0-ff81-4c19-889b-340a2013c07a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:32.988778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-73d87ad0-ff81-4c19-889b-340a2013c07a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:32.990063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-73d87ad0-ff81-4c19-889b-340a2013c07a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:33.051874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:33.093392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 507/999] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:33.395989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa549ca-7a0d-47d9-a136-61b23f7e2008 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:33.396266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa549ca-7a0d-47d9-a136-61b23f7e2008 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:33.396532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa549ca-7a0d-47d9-a136-61b23f7e2008 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:33.397910 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 493/1000] 10.222.0.23 () {54 vars in 964 bytes} [Wed Oct 12 02:21:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:33.403851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c25cc-68d0-4588-9f08-be3905d02b31 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:33.404738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c25cc-68d0-4588-9f08-be3905d02b31 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:33.405011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c25cc-68d0-4588-9f08-be3905d02b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:33.482254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a82c25cc-68d0-4588-9f08-be3905d02b31 None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:33.483553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a82c25cc-68d0-4588-9f08-be3905d02b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:33.558903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 508/1001] 10.222.0.23 () {58 vars in 1061 bytes} [Wed Oct 12 02:21:33 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:33.568147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa549ca-7a0d-47d9-a136-61b23f7e2008 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:33.569704 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1fa549ca-7a0d-47d9-a136-61b23f7e2008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:33.589498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:33.672167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa549ca-7a0d-47d9-a136-61b23f7e2008 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:33.745689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:33.746298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:33.746575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:33.746847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:33.748974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:33.821736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:33.858826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:33.860268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7031f6-6ae8-4dce-bf66-425a6783881a service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:33.879792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 494/1002] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:36.295060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe9a99-1725-44e5-81f9-b293d58713c0 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:36.295579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe9a99-1725-44e5-81f9-b293d58713c0 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:36.295579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe9a99-1725-44e5-81f9-b293d58713c0 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:36.297515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 509/1003] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:36.301940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020cfee5-4405-4781-9626-096603dc1fde None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:36.302327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020cfee5-4405-4781-9626-096603dc1fde None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:36.302469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020cfee5-4405-4781-9626-096603dc1fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:36.369126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-020cfee5-4405-4781-9626-096603dc1fde None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:36.370333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-020cfee5-4405-4781-9626-096603dc1fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:36.465666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 495/1004] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:36.749803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe9a99-1725-44e5-81f9-b293d58713c0 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:36.751392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-29fe9a99-1725-44e5-81f9-b293d58713c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:36.812055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:36.856313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe9a99-1725-44e5-81f9-b293d58713c0 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:36.930771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:36.931403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:36.931679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:36.931964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:36.933554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:37.042370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:37.045006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:37.046837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b38215a-e205-48a0-a109-93f817263c54 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:37.067039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 510/1005] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:21:39.288376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2362f62-a7fc-4554-af97-21c316b250e0 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:39.288933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2362f62-a7fc-4554-af97-21c316b250e0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:39.288933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2362f62-a7fc-4554-af97-21c316b250e0 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:39.290289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 496/1006] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:39.295773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a380f3b-9c61-4e59-b90d-b1e70ca27205 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:39.296192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a380f3b-9c61-4e59-b90d-b1e70ca27205 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:39.296300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a380f3b-9c61-4e59-b90d-b1e70ca27205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:39.370205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5a380f3b-9c61-4e59-b90d-b1e70ca27205 None None] MFA Rules not processed for user `e66d8b7ddd4d4982adf0584a255beaf1`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:39.371421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5a380f3b-9c61-4e59-b90d-b1e70ca27205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:39.468807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 511/1007] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:39 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:39.755774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2362f62-a7fc-4554-af97-21c316b250e0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:39.757426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a2362f62-a7fc-4554-af97-21c316b250e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:39.787953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:39.861808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2362f62-a7fc-4554-af97-21c316b250e0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:39.936170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:39.936815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:39.937084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:39.937352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:39.938939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:40.019152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:40.055873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:40.057586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a5d29e-9828-4e14-8fc7-3c7fbc0a90e4 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:40.077075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 497/1008] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:39 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:42.426930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dbeab0-5722-4beb-9ba2-639cac2684df None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:42.427428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dbeab0-5722-4beb-9ba2-639cac2684df None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:42.427428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dbeab0-5722-4beb-9ba2-639cac2684df None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:42.428783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 512/1009] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:42.433981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8644b373-276e-4e49-9387-634adc7c610b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:42.434389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8644b373-276e-4e49-9387-634adc7c610b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:42.434518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8644b373-276e-4e49-9387-634adc7c610b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:42.505418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8644b373-276e-4e49-9387-634adc7c610b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:42.506631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8644b373-276e-4e49-9387-634adc7c610b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:42.598986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 498/1010] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:42.883311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dbeab0-5722-4beb-9ba2-639cac2684df None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:42.884866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-53dbeab0-5722-4beb-9ba2-639cac2684df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:42.953077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:42.986857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dbeab0-5722-4beb-9ba2-639cac2684df None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:43.057634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:43.058230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:43.058503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:43.058768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:43.060367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:43.168846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:43.170315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef01c1f-22bf-4180-a126-75f36db9f998 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:43.180037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:43.188727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 513/1011] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:45.359475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b248043e-1022-492e-97e4-ba22af84dc31 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:45.360022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b248043e-1022-492e-97e4-ba22af84dc31 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:45.360022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b248043e-1022-492e-97e4-ba22af84dc31 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:45.361412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 499/1012] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:45.367021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58617660-e61c-48ba-9247-3cc27cbd671d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:45.367425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58617660-e61c-48ba-9247-3cc27cbd671d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:45.367522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58617660-e61c-48ba-9247-3cc27cbd671d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:45.435305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-58617660-e61c-48ba-9247-3cc27cbd671d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:45.436562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-58617660-e61c-48ba-9247-3cc27cbd671d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:45.530151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 514/1013] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:45.815482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b248043e-1022-492e-97e4-ba22af84dc31 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:45.817068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b248043e-1022-492e-97e4-ba22af84dc31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:45.861474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:45.925110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b248043e-1022-492e-97e4-ba22af84dc31 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:45.997401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:45.998009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:45.998284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:45.998552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:46.000134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:46.081934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:46.108477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:46.109939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65bef78-9403-47a7-9ae3-d597023d2ace service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:46.127884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 500/1014] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:48.582647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a7e6e-fbda-4e17-a5b3-0ce4b9a0b7ce None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:48.583206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a7e6e-fbda-4e17-a5b3-0ce4b9a0b7ce None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:48.583206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a7e6e-fbda-4e17-a5b3-0ce4b9a0b7ce None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:48.584511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 515/1015] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:48.589771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec036a-0d19-423a-b994-a3db4e7c1f1b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:48.590187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec036a-0d19-423a-b994-a3db4e7c1f1b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:48.590286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec036a-0d19-423a-b994-a3db4e7c1f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:48.654493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-afec036a-0d19-423a-b994-a3db4e7c1f1b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:48.655678 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-afec036a-0d19-423a-b994-a3db4e7c1f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:48.754950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 501/1016] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:49.037872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a7e6e-fbda-4e17-a5b3-0ce4b9a0b7ce None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:49.039427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-493a7e6e-fbda-4e17-a5b3-0ce4b9a0b7ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:49.117467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:49.144627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a7e6e-fbda-4e17-a5b3-0ce4b9a0b7ce None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:49.218779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:49.219386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:49.219661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:49.219930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:49.221549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:49.328064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:49.329537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b427f7-97be-4c1d-b617-038104b771b8 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:49.347533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 516/1017] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:49 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:53.336106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ee9b5d-ed60-48fe-9923-4a20f0ff5a3f None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:53.336631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ee9b5d-ed60-48fe-9923-4a20f0ff5a3f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:53.336631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ee9b5d-ed60-48fe-9923-4a20f0ff5a3f None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:53.338184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 502/1018] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:53.343697 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b05d1-bf00-4f4b-a216-ffd72601cb2b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:53.344142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b05d1-bf00-4f4b-a216-ffd72601cb2b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:53.344239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b05d1-bf00-4f4b-a216-ffd72601cb2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:53.349852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:53.413169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-611b05d1-bf00-4f4b-a216-ffd72601cb2b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:53.414306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-611b05d1-bf00-4f4b-a216-ffd72601cb2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:53.510323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 517/1019] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:53.794730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ee9b5d-ed60-48fe-9923-4a20f0ff5a3f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:53.796260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d8ee9b5d-ed60-48fe-9923-4a20f0ff5a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:53.844388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:53.898424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ee9b5d-ed60-48fe-9923-4a20f0ff5a3f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:53.971181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:53.971780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:53.972070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:53.972322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:53.973930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:54.064384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:54.081083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:54.082521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568111b7-e98d-47b0-b8b9-ed2dff3d9704 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:54.103832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 503/1020] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:21:56.891510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336bb738-9108-4820-a18a-5f2f2e235a94 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:56.892025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336bb738-9108-4820-a18a-5f2f2e235a94 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:56.892025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336bb738-9108-4820-a18a-5f2f2e235a94 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:56.893422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 518/1021] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:21:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:21:56.898668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d8b2ad-0249-4261-882b-9fdbdef084c9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:56.899088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d8b2ad-0249-4261-882b-9fdbdef084c9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:56.899197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d8b2ad-0249-4261-882b-9fdbdef084c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:56.967225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-58d8b2ad-0249-4261-882b-9fdbdef084c9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:21:56.968412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-58d8b2ad-0249-4261-882b-9fdbdef084c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:57.064995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 504/1022] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:21:56 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:21:57.345811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336bb738-9108-4820-a18a-5f2f2e235a94 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:21:57.347361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-336bb738-9108-4820-a18a-5f2f2e235a94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:57.437555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:21:57.457796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336bb738-9108-4820-a18a-5f2f2e235a94 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:21:57.533734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:21:57.534345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:21:57.534624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:21:57.534896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:21:57.536491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:21:57.645879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:21:57.647251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de243b-3db8-45ce-84a9-216cf6c402d6 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:21:57.666199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 519/1023] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:21:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:01.771293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f976850-9907-4617-83b7-9d89745c38f9 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:01.771732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f976850-9907-4617-83b7-9d89745c38f9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:01.771823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f976850-9907-4617-83b7-9d89745c38f9 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:01.773220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 505/1024] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:01.779243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e0fe3a-7142-4b07-aac7-d0c81d39efde None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:01.779729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e0fe3a-7142-4b07-aac7-d0c81d39efde None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:01.779863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e0fe3a-7142-4b07-aac7-d0c81d39efde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:01.796616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:01.850443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-62e0fe3a-7142-4b07-aac7-d0c81d39efde None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:01.851667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-62e0fe3a-7142-4b07-aac7-d0c81d39efde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:01.947674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 520/1025] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:02.232185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f976850-9907-4617-83b7-9d89745c38f9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:02.233748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6f976850-9907-4617-83b7-9d89745c38f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:02.291412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:02.336601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f976850-9907-4617-83b7-9d89745c38f9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:02.409277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:02.409809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:02.410077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:02.410336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:02.411903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:02.513917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:02.517533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:02.519107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c388bd-61c9-4966-b157-cf2e387a4705 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:02.536929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 506/1026] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:07.442489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d60cfb9-7ce6-4323-97ea-7453ef81be28 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:07.443004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d60cfb9-7ce6-4323-97ea-7453ef81be28 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:07.443561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d60cfb9-7ce6-4323-97ea-7453ef81be28 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:07.444673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 521/1027] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:22:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:07.451214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fe808-8beb-4d09-8338-5d858b275e33 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:07.451650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fe808-8beb-4d09-8338-5d858b275e33 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:07.451739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fe808-8beb-4d09-8338-5d858b275e33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:07.556096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-477fe808-8beb-4d09-8338-5d858b275e33 None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:07.558757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-477fe808-8beb-4d09-8338-5d858b275e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:07.721342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 507/1028] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:22:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:11.946115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=59 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:11.964964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e2ecf3-ab3d-44ed-b6e2-c03a1c7b0bd2 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:11.965367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e2ecf3-ab3d-44ed-b6e2-c03a1c7b0bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:11.965745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e2ecf3-ab3d-44ed-b6e2-c03a1c7b0bd2 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:11.967171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 522/1029] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:22:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:11.972971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e234de7-ac50-4954-82f2-25390fcb5181 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:11.973444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e234de7-ac50-4954-82f2-25390fcb5181 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:11.973752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e234de7-ac50-4954-82f2-25390fcb5181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:12.030343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:12.054469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2e234de7-ac50-4954-82f2-25390fcb5181 None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:12.055818 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2e234de7-ac50-4954-82f2-25390fcb5181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:12.183112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 508/1030] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:22:11 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:13.430250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8be7c-04e6-46bf-b616-5fd4a4d822f9 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:13.430831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8be7c-04e6-46bf-b616-5fd4a4d822f9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:13.430831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8be7c-04e6-46bf-b616-5fd4a4d822f9 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:13.432585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 523/1031] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:22:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:13.437394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea69444-94ce-462e-b709-090f0adf1775 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:13.437841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea69444-94ce-462e-b709-090f0adf1775 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:13.437944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea69444-94ce-462e-b709-090f0adf1775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:13.516087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-eea69444-94ce-462e-b709-090f0adf1775 None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:13.517335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-eea69444-94ce-462e-b709-090f0adf1775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:13.562397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:13.627557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 509/1032] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:22:13 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:17.136717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880907a0-85be-4644-8881-0e11bfa7d8b8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:17.137563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880907a0-85be-4644-8881-0e11bfa7d8b8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:17.138001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880907a0-85be-4644-8881-0e11bfa7d8b8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:17.144004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 524/1033] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:17.146025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68f6ba-5286-4964-a0e0-d29a7ac509d4 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:17.147041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68f6ba-5286-4964-a0e0-d29a7ac509d4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:17.147511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68f6ba-5286-4964-a0e0-d29a7ac509d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:17.223008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ad68f6ba-5286-4964-a0e0-d29a7ac509d4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:17.224326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ad68f6ba-5286-4964-a0e0-d29a7ac509d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:17.332746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 510/1034] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:17.631165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f3c60-1fdc-4a15-a4a7-174a5fd325e2 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:17.631518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f3c60-1fdc-4a15-a4a7-174a5fd325e2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:17.631855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f3c60-1fdc-4a15-a4a7-174a5fd325e2 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:17.633520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 525/1035] 10.222.0.23 () {54 vars in 961 bytes} [Wed Oct 12 02:22:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:17.640170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861e166d-22c9-417c-ac7d-82d8b96ed3c0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:17.640456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861e166d-22c9-417c-ac7d-82d8b96ed3c0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:17.640748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861e166d-22c9-417c-ac7d-82d8b96ed3c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:17.660765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:17.724737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-861e166d-22c9-417c-ac7d-82d8b96ed3c0 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:17.725870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-861e166d-22c9-417c-ac7d-82d8b96ed3c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:17.847925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 511/1036] 10.222.0.23 () {58 vars in 1058 bytes} [Wed Oct 12 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:17.855719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f3c60-1fdc-4a15-a4a7-174a5fd325e2 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:17.857273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-031f3c60-1fdc-4a15-a4a7-174a5fd325e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:18.020676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f3c60-1fdc-4a15-a4a7-174a5fd325e2 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:18.102244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:18.145153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:18.145804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:18.146074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:18.146383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:18.148019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:18.247912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:18.249683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8cd13b-e62f-47f6-a63c-756d9e5808f8 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:18.271284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 526/1037] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:20.633166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b176141-aba6-4db0-9d60-9024cda2398a None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:20.633702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b176141-aba6-4db0-9d60-9024cda2398a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:20.633785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b176141-aba6-4db0-9d60-9024cda2398a None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:20.636740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 512/1038] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:20.640845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a20104-7050-40bb-ac1a-59043d0af14e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:20.641281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a20104-7050-40bb-ac1a-59043d0af14e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:20.641386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a20104-7050-40bb-ac1a-59043d0af14e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:20.750818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b3a20104-7050-40bb-ac1a-59043d0af14e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:20.752814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b3a20104-7050-40bb-ac1a-59043d0af14e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:20.794078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:20.863129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 527/1039] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:21.153619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b176141-aba6-4db0-9d60-9024cda2398a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:21.155276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0b176141-aba6-4db0-9d60-9024cda2398a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:21.229232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:21.330460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b176141-aba6-4db0-9d60-9024cda2398a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:21.461668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:21.461668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:21.461668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:21.462596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:21.464933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:21.489060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:21.564554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:21.566116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d8fb37-3237-4d8f-8d9c-4d7a0f5c625c service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:21.584604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 513/1040] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:24.157336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915c550-f677-4cfb-bb78-c8ecf1e92eb8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:24.157881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915c550-f677-4cfb-bb78-c8ecf1e92eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:24.157881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915c550-f677-4cfb-bb78-c8ecf1e92eb8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:24.159225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 528/1041] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:24.164563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c738a4-4089-4c21-a4b8-75af65439299 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:24.164980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c738a4-4089-4c21-a4b8-75af65439299 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:24.165146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c738a4-4089-4c21-a4b8-75af65439299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:24.234084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-48c738a4-4089-4c21-a4b8-75af65439299 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:24.235327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-48c738a4-4089-4c21-a4b8-75af65439299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:24.336444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:24.345128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 514/1042] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:24.627868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e042b83-1fa2-4d03-84af-4231f1596be2 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:24.628152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e042b83-1fa2-4d03-84af-4231f1596be2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:24.628422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e042b83-1fa2-4d03-84af-4231f1596be2 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:24.629795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 529/1043] 10.222.0.23 () {54 vars in 961 bytes} [Wed Oct 12 02:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:24.635794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad4aea7-c98d-42e7-82d3-1a172e5c7195 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:24.636080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad4aea7-c98d-42e7-82d3-1a172e5c7195 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:24.636357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad4aea7-c98d-42e7-82d3-1a172e5c7195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:24.716166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1ad4aea7-c98d-42e7-82d3-1a172e5c7195 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:24.717432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1ad4aea7-c98d-42e7-82d3-1a172e5c7195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:24.844525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 515/1044] 10.222.0.23 () {58 vars in 1058 bytes} [Wed Oct 12 02:22:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:24.852144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e042b83-1fa2-4d03-84af-4231f1596be2 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:24.854255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0e042b83-1fa2-4d03-84af-4231f1596be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:25.008472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e042b83-1fa2-4d03-84af-4231f1596be2 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:25.029620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:25.133332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:25.133938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:25.134205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:25.134463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:25.136566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:25.236037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:25.237683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f406be-9212-4141-9129-ef09bcbf42d1 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:25.256762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 530/1045] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:25.319059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066781c3-53dd-4603-b582-3d6e510481f7 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:25.319541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066781c3-53dd-4603-b582-3d6e510481f7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:25.319622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066781c3-53dd-4603-b582-3d6e510481f7 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:25.321101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 516/1046] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:25.326872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5b02f4-cd10-4611-8534-e86af1fd9164 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:25.327153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5b02f4-cd10-4611-8534-e86af1fd9164 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:25.327424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5b02f4-cd10-4611-8534-e86af1fd9164 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:25.363828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:25.409668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-2f5b02f4-cd10-4611-8534-e86af1fd9164 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:25.411094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2f5b02f4-cd10-4611-8534-e86af1fd9164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:25.532830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 531/1047] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:25.864291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec397bb1-95aa-40dd-8655-bb9d9da88b53 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:25.864291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec397bb1-95aa-40dd-8655-bb9d9da88b53 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:25.864291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec397bb1-95aa-40dd-8655-bb9d9da88b53 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:25.865941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 517/1048] 10.222.0.23 () {54 vars in 963 bytes} [Wed Oct 12 02:22:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:25.871509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44e4b5-3200-445b-b91b-a8898e7733cf None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:25.871896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44e4b5-3200-445b-b91b-a8898e7733cf None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:25.872362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44e4b5-3200-445b-b91b-a8898e7733cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:25.951761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ec44e4b5-3200-445b-b91b-a8898e7733cf None None] MFA Rules not processed for user `575abe426dae4b868bb1c35537a9be35`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:25.953148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ec44e4b5-3200-445b-b91b-a8898e7733cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:26.002656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:26.035332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 532/1049] 10.222.0.23 () {58 vars in 1060 bytes} [Wed Oct 12 02:22:25 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:26.043579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec397bb1-95aa-40dd-8655-bb9d9da88b53 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:26.045236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ec397bb1-95aa-40dd-8655-bb9d9da88b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:26.098572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:26.158205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec397bb1-95aa-40dd-8655-bb9d9da88b53 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:26.238781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:26.239403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:26.239937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:26.240203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:26.242264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:26.351753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:26.367080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:26.368618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e37518b-92f3-4a2f-872f-7f71a428e1df service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:26.390445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 518/1050] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:22:27.492802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fb3dc-8548-4034-b515-775c48da197b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:27.493330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fb3dc-8548-4034-b515-775c48da197b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:27.493420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fb3dc-8548-4034-b515-775c48da197b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:27.495327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 533/1051] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:27.500622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dd3aef-7435-4606-9737-f0999e033800 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:27.501057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dd3aef-7435-4606-9737-f0999e033800 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:27.501327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dd3aef-7435-4606-9737-f0999e033800 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:27.568076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-75dd3aef-7435-4606-9737-f0999e033800 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:27.569307 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-75dd3aef-7435-4606-9737-f0999e033800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:27.664807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 519/1052] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:27.962045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fb3dc-8548-4034-b515-775c48da197b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:27.963631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-728fb3dc-8548-4034-b515-775c48da197b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:28.119582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fb3dc-8548-4034-b515-775c48da197b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:28.174505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:28.238185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:28.238866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:28.239204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:28.239535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:28.241459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:28.338883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:28.340483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efae37ee-6a1a-4f9b-bfb0-46fcd8e589aa service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:28.359272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 534/1053] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:30.547666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccfd75-97d3-492d-9b0c-1241a8904421 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:30.548200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccfd75-97d3-492d-9b0c-1241a8904421 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:30.548200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccfd75-97d3-492d-9b0c-1241a8904421 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:30.549868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 520/1054] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:30.555008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d101c38-d63d-4b41-bbef-5f7ab50fd021 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:30.555428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d101c38-d63d-4b41-bbef-5f7ab50fd021 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:30.555602 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d101c38-d63d-4b41-bbef-5f7ab50fd021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:30.623003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3d101c38-d63d-4b41-bbef-5f7ab50fd021 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:30.624193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3d101c38-d63d-4b41-bbef-5f7ab50fd021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:30.631007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:30.732559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 535/1055] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:31.026765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccfd75-97d3-492d-9b0c-1241a8904421 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:31.028661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-61ccfd75-97d3-492d-9b0c-1241a8904421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:31.129549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:31.225756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccfd75-97d3-492d-9b0c-1241a8904421 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:31.343037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:31.343648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:31.344024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:31.344306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:31.345915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:31.366448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:31.443476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:31.445346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2090bbe2-360e-485f-a2cb-3564709223c4 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:31.465114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 521/1056] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:33.671209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd0820-914c-4e74-bb27-6b90799ae2f8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:33.671658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd0820-914c-4e74-bb27-6b90799ae2f8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:33.671772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd0820-914c-4e74-bb27-6b90799ae2f8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:33.673345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 536/1057] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:33.678544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6504c-07c4-486e-8462-4af7ec6af6e8 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:33.678963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6504c-07c4-486e-8462-4af7ec6af6e8 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:33.679165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6504c-07c4-486e-8462-4af7ec6af6e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:33.784631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1fe6504c-07c4-486e-8462-4af7ec6af6e8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:33.786960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1fe6504c-07c4-486e-8462-4af7ec6af6e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:33.898291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:33.904434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 522/1058] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:34.380947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcd0820-914c-4e74-bb27-6b90799ae2f8 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:34.382473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2bcd0820-914c-4e74-bb27-6b90799ae2f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:34.538508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:34.550619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcd0820-914c-4e74-bb27-6b90799ae2f8 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:34.671297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:34.672330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:34.672330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:34.672330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:34.673943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:34.780397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:34.783701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66e2be6-e07b-4982-9517-998e43d2dd80 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:34.802644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:34.820540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 537/1059] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:37.464640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7547d02-c8e3-4364-81a7-276617acf982 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:37.465298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7547d02-c8e3-4364-81a7-276617acf982 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:37.465298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7547d02-c8e3-4364-81a7-276617acf982 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:37.466562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 523/1060] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:37.472052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b097034b-d90d-4c17-b94e-cf0090a724c7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:37.472523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b097034b-d90d-4c17-b94e-cf0090a724c7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:37.472619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b097034b-d90d-4c17-b94e-cf0090a724c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:37.546552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b097034b-d90d-4c17-b94e-cf0090a724c7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:37.547778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b097034b-d90d-4c17-b94e-cf0090a724c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:37.651229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 538/1061] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:37.937948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7547d02-c8e3-4364-81a7-276617acf982 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:37.939501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b7547d02-c8e3-4364-81a7-276617acf982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:38.086732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7547d02-c8e3-4364-81a7-276617acf982 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:38.175095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:38.222433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:38.223063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:38.223295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:38.223571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:38.225220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:38.316359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:38.318025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244d1b12-96f1-4b6a-9171-a72cfeb44f75 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:38.335921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 524/1062] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:40.505083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b809bc-ffee-44a4-acf7-52c9f216e100 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:40.505753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b809bc-ffee-44a4-acf7-52c9f216e100 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:40.505753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b809bc-ffee-44a4-acf7-52c9f216e100 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:40.507656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 539/1063] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:40.514652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b5e64-4c58-4bd7-aab3-b484ad4802b4 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:40.515134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b5e64-4c58-4bd7-aab3-b484ad4802b4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:40.515366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b5e64-4c58-4bd7-aab3-b484ad4802b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:40.587504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b65b5e64-4c58-4bd7-aab3-b484ad4802b4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:40.588811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b65b5e64-4c58-4bd7-aab3-b484ad4802b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:40.622804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:40.688501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 525/1064] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:40.984884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b809bc-ffee-44a4-acf7-52c9f216e100 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:40.986564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-82b809bc-ffee-44a4-acf7-52c9f216e100 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:41.067370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:41.143129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b809bc-ffee-44a4-acf7-52c9f216e100 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:41.259034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:41.259647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:41.259923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:41.260191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:41.261818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:41.301583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:41.354885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:41.357067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e61ca-911e-4e4d-b878-ca395b8007ba service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:41.375057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 540/1065] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:42.925401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400dc0d9-aee5-43e4-879f-bf38c5396c11 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:42.925953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400dc0d9-aee5-43e4-879f-bf38c5396c11 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:42.925953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400dc0d9-aee5-43e4-879f-bf38c5396c11 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:42.927344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 526/1066] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:42.933273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a841ee1a-08dc-42a7-af14-3b5ec75d3e9e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:42.933709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a841ee1a-08dc-42a7-af14-3b5ec75d3e9e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:42.933897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a841ee1a-08dc-42a7-af14-3b5ec75d3e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:43.001551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a841ee1a-08dc-42a7-af14-3b5ec75d3e9e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:43.002777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a841ee1a-08dc-42a7-af14-3b5ec75d3e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:43.104717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 541/1067] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:43.404977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c0c789-ae27-4574-a3e9-2b6f8e76df9b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:43.405261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c0c789-ae27-4574-a3e9-2b6f8e76df9b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:43.405532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c0c789-ae27-4574-a3e9-2b6f8e76df9b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:43.406861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 527/1068] 10.222.0.23 () {54 vars in 962 bytes} [Wed Oct 12 02:22:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:43.412430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:43.412747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:43.413020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:43.429488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:43.498441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] MFA Rules not processed for user `3d2788633742480981376ed8b171ac89`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:43.499663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:43.563490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26eb891-ec02-474c-8653-84fb104f7dc6 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:43.563698 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26eb891-ec02-474c-8653-84fb104f7dc6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:43.563940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26eb891-ec02-474c-8653-84fb104f7dc6 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:43.565318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 528/1069] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:22:43.570299 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:43.570576 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:43.570846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:43.577779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 542/1070] 10.222.0.23 () {58 vars in 1059 bytes} [Wed Oct 12 02:22:43 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:43.587430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:43.595016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1bbe2d-6682-4f4c-89b2-7cb649d3d53c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:43.637763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:43.638982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:43.673689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:43.674355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:43.674631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:43.674897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:43.676499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:43.706762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:43.741338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 529/1071] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:22:43.792410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:43.793903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95be0951-2569-4faa-8b75-9979adf7d2b2 service glance] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:43.812785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 543/1072] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:22:44.040934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:44.042164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:44.057258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:44.164443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdea852-92bb-4ac8-a5d8-9f6ce134e17d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:44.296468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:44.297020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:44.297296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:44.297565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:44.299155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:44.308467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:44.397509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:44.399723 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a511d67e-9960-4f0d-b055-c02dfc50d85d service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:44.420974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 530/1073] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:44.568184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e35d610-ad86-4ab3-8283-8a45f9245537 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:44.568819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e35d610-ad86-4ab3-8283-8a45f9245537 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:44.568819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e35d610-ad86-4ab3-8283-8a45f9245537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:44.646166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-8e35d610-ad86-4ab3-8283-8a45f9245537 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:44.647403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8e35d610-ad86-4ab3-8283-8a45f9245537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:44.735684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:44.742022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 544/1074] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:22:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:44.789242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09a518-19b8-42c2-afd3-03710cee780d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:44.789601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09a518-19b8-42c2-afd3-03710cee780d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:44.790657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09a518-19b8-42c2-afd3-03710cee780d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:44.792126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 531/1075] 10.222.0.23 () {54 vars in 943 bytes} [Wed Oct 12 02:22:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:44.798920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec10573-25df-498c-93d3-a39db0b5a94c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:44.799206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec10573-25df-498c-93d3-a39db0b5a94c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:44.799480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec10573-25df-498c-93d3-a39db0b5a94c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:44.878051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-dec10573-25df-498c-93d3-a39db0b5a94c None None] MFA Rules not processed for user `c57115b9feff40629f5081e32ff4bad3`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:44.879245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dec10573-25df-498c-93d3-a39db0b5a94c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:44.990228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 545/1076] 10.222.0.23 () {58 vars in 1040 bytes} [Wed Oct 12 02:22:44 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:22:45.000130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09a518-19b8-42c2-afd3-03710cee780d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:45.001716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2a09a518-19b8-42c2-afd3-03710cee780d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:45.139665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:45.140996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09a518-19b8-42c2-afd3-03710cee780d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:45.246393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:45.246991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:45.247269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:45.247539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:45.249196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:45.330946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:45.332600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80daef6-0355-4599-b1e4-2bb003c7dc7c service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:45.341151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 532/1077] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:22:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:22:46.649097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f460f517-0a26-401e-8986-74e4d161247d None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:46.649638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f460f517-0a26-401e-8986-74e4d161247d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:46.649638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f460f517-0a26-401e-8986-74e4d161247d None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:46.650972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 546/1078] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:46.656253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de72794d-b597-4d54-ac51-cf529e39ec66 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:46.656772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de72794d-b597-4d54-ac51-cf529e39ec66 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:46.656858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de72794d-b597-4d54-ac51-cf529e39ec66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:46.701774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:46.731072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-de72794d-b597-4d54-ac51-cf529e39ec66 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:46.731419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-de72794d-b597-4d54-ac51-cf529e39ec66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:46.876788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 533/1079] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:47.165616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f460f517-0a26-401e-8986-74e4d161247d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:47.167150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f460f517-0a26-401e-8986-74e4d161247d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:47.231939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:47.326896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f460f517-0a26-401e-8986-74e4d161247d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:47.460711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:47.461346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:47.462150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:47.462411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:47.464022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:47.482818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:47.566607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:47.568231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecc7682-666f-4d54-b1da-fe9b191dfaf2 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:47.596899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 547/1080] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:49.789392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794b86b-6638-49a5-8b21-0384799c19c5 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:49.790067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794b86b-6638-49a5-8b21-0384799c19c5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:49.790067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794b86b-6638-49a5-8b21-0384799c19c5 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:49.791463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 534/1081] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:49.796721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62dd94-0047-4da6-ab78-acf3c317aa7e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:49.797200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62dd94-0047-4da6-ab78-acf3c317aa7e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:49.797288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62dd94-0047-4da6-ab78-acf3c317aa7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:49.869447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-4e62dd94-0047-4da6-ab78-acf3c317aa7e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:49.870734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4e62dd94-0047-4da6-ab78-acf3c317aa7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:49.967215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:49.974784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 548/1082] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:22:50.263402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8794b86b-6638-49a5-8b21-0384799c19c5 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:50.265615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8794b86b-6638-49a5-8b21-0384799c19c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:50.380316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:50.429778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8794b86b-6638-49a5-8b21-0384799c19c5 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:50.552002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:50.552623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:50.552927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:50.553194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:50.554798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:50.635973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:50.656846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:50.658608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89402098-a364-49c2-9a80-dd462abd3906 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:50.678788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 535/1083] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:22:52.953449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094ceb0-96de-42dc-9d05-6fe0bfef49c9 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:52.953996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094ceb0-96de-42dc-9d05-6fe0bfef49c9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:52.953996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094ceb0-96de-42dc-9d05-6fe0bfef49c9 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:52.955519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 549/1084] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:22:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:22:52.961142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e0ed7-13fe-471f-aa63-c5125e99e20a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:52.962099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e0ed7-13fe-471f-aa63-c5125e99e20a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:52.962203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e0ed7-13fe-471f-aa63-c5125e99e20a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:53.034034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0b1e0ed7-13fe-471f-aa63-c5125e99e20a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:22:53.035279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0b1e0ed7-13fe-471f-aa63-c5125e99e20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:53.151285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 536/1085] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:22:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:22:53.444734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094ceb0-96de-42dc-9d05-6fe0bfef49c9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:22:53.446337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5094ceb0-96de-42dc-9d05-6fe0bfef49c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:53.608299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094ceb0-96de-42dc-9d05-6fe0bfef49c9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:22:53.683426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:22:53.741584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:22:53.742438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:22:53.742803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:22:53.743207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:22:53.745153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:22:53.846752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:22:53.848362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dde1d-db8d-4fd8-8ed1-5d6e50fd9879 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:22:53.868741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 550/1086] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:22:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:03.149588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=78 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:03.866350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=65 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:07.983344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2561b6-eb26-4306-885f-669ca0c93d3d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:07.983888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2561b6-eb26-4306-885f-669ca0c93d3d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:07.983888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2561b6-eb26-4306-885f-669ca0c93d3d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:07.985276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 537/1087] 10.222.0.23 () {54 vars in 918 bytes} [Wed Oct 12 02:23:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:07.990709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76822556-5721-43a5-9481-1f01be87c9bc None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:07.991129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76822556-5721-43a5-9481-1f01be87c9bc None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:07.991232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76822556-5721-43a5-9481-1f01be87c9bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:08.066890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-76822556-5721-43a5-9481-1f01be87c9bc None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:08.068086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-76822556-5721-43a5-9481-1f01be87c9bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:08.174249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 551/1088] 10.222.0.23 () {58 vars in 1015 bytes} [Wed Oct 12 02:23:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:08.188341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831478d-4c9f-4b15-8f72-774b1a386319 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:08.188650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831478d-4c9f-4b15-8f72-774b1a386319 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:08.188951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831478d-4c9f-4b15-8f72-774b1a386319 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:08.190271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 538/1089] 10.222.0.23 () {54 vars in 918 bytes} [Wed Oct 12 02:23:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:08.194755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231c966-ab03-4d81-81c5-208810266c68 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:08.195558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231c966-ab03-4d81-81c5-208810266c68 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:08.195829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4231c966-ab03-4d81-81c5-208810266c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:08.268614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:08.271659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-4231c966-ab03-4d81-81c5-208810266c68 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:08.273441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4231c966-ab03-4d81-81c5-208810266c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:08.388804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 552/1090] 10.222.0.23 () {58 vars in 1015 bytes} [Wed Oct 12 02:23:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:10.944270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac23ee7-7d63-4264-ade7-defd9c2fd6b0 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:10.945278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac23ee7-7d63-4264-ade7-defd9c2fd6b0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:10.945408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac23ee7-7d63-4264-ade7-defd9c2fd6b0 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:10.946684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 539/1091] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:10.952012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9115542a-f13e-41c4-9e51-66769568d998 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:10.952429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9115542a-f13e-41c4-9e51-66769568d998 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:10.952548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9115542a-f13e-41c4-9e51-66769568d998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:11.018816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-9115542a-f13e-41c4-9e51-66769568d998 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:11.020018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9115542a-f13e-41c4-9e51-66769568d998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:11.074389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:11.115988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 553/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:11.404995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac23ee7-7d63-4264-ade7-defd9c2fd6b0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:11.406573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cac23ee7-7d63-4264-ade7-defd9c2fd6b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:11.548879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac23ee7-7d63-4264-ade7-defd9c2fd6b0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:11.620351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:11.660280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:11.660845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:11.661128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:11.661417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:11.662991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:11.749994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:11.751562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c226491e-88f8-4246-9a83-4bc73274f793 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:11.769491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 540/1093] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:12.898782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cee378-b9e6-4ba2-a327-d3cafbc66df8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:12.899242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cee378-b9e6-4ba2-a327-d3cafbc66df8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:12.899354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cee378-b9e6-4ba2-a327-d3cafbc66df8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:12.900723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 554/1094] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:12.907076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53854750-99f0-4d0d-8439-c40614d47b3b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:12.907520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53854750-99f0-4d0d-8439-c40614d47b3b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:12.907631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53854750-99f0-4d0d-8439-c40614d47b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:12.982075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-53854750-99f0-4d0d-8439-c40614d47b3b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:12.983264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-53854750-99f0-4d0d-8439-c40614d47b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:13.010902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:13.080894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 541/1095] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:13.231785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cee378-b9e6-4ba2-a327-d3cafbc66df8 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:13.233379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a8cee378-b9e6-4ba2-a327-d3cafbc66df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:13.378566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cee378-b9e6-4ba2-a327-d3cafbc66df8 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:13.418128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:13.490023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:13.490554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:13.490827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:13.491090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:13.493201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:13.581581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:13.583153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae4e37-5945-4ca4-ad29-e92546f2ba2e service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:13.600493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 555/1096] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:14.724534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb650a30-9800-423b-8407-5ff3d5a94604 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:14.725100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb650a30-9800-423b-8407-5ff3d5a94604 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:14.725100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb650a30-9800-423b-8407-5ff3d5a94604 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:14.726449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 542/1097] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:14.732625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1bd2e-f951-4c6f-8e29-90aefe44046c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:14.733073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1bd2e-f951-4c6f-8e29-90aefe44046c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:14.733254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1bd2e-f951-4c6f-8e29-90aefe44046c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:14.788332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:14.809880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-cdf1bd2e-f951-4c6f-8e29-90aefe44046c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:14.811082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cdf1bd2e-f951-4c6f-8e29-90aefe44046c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:14.906505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 556/1098] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:15.059098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb650a30-9800-423b-8407-5ff3d5a94604 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:15.060526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bb650a30-9800-423b-8407-5ff3d5a94604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:15.204423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb650a30-9800-423b-8407-5ff3d5a94604 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:15.216317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:15.315937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:15.316458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:15.316753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:15.317021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:15.318601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:15.405930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:15.407497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40366d96-4ad3-45b6-ba3e-d530d716bdf6 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:15.425387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 543/1099] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:16.525532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc09b1-7476-456b-99c6-3f3b19ef6a59 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:16.526068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc09b1-7476-456b-99c6-3f3b19ef6a59 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:16.526068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc09b1-7476-456b-99c6-3f3b19ef6a59 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:16.527448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 557/1100] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:16.533457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a188fd4-67ba-4d8f-8a05-64b35265ed74 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:16.533847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a188fd4-67ba-4d8f-8a05-64b35265ed74 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:16.533997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a188fd4-67ba-4d8f-8a05-64b35265ed74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:16.553984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:16.610871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2a188fd4-67ba-4d8f-8a05-64b35265ed74 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:16.612078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2a188fd4-67ba-4d8f-8a05-64b35265ed74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:16.705383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 544/1101] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:16.855857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc09b1-7476-456b-99c6-3f3b19ef6a59 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:16.857361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a2bc09b1-7476-456b-99c6-3f3b19ef6a59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:16.967742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:17.003180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc09b1-7476-456b-99c6-3f3b19ef6a59 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:17.115259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:17.115783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:17.116052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:17.116336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:17.117959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:17.194096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:17.208745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:17.210973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c054ca-1a33-4a9d-9ea1-d6f1afa34d0b service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:17.228488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 558/1102] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:18.329007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91dda1-2c55-4000-90de-299660795719 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:18.329573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91dda1-2c55-4000-90de-299660795719 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:18.329573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91dda1-2c55-4000-90de-299660795719 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:18.330874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 545/1103] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:18.337420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee409c6d-abc8-470b-8185-737bcae22914 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:18.337858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee409c6d-abc8-470b-8185-737bcae22914 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:18.337953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee409c6d-abc8-470b-8185-737bcae22914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:18.414103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ee409c6d-abc8-470b-8185-737bcae22914 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:18.415245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ee409c6d-abc8-470b-8185-737bcae22914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:18.510546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 559/1104] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:18.660654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91dda1-2c55-4000-90de-299660795719 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:18.662174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ff91dda1-2c55-4000-90de-299660795719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:18.746652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:18.807301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91dda1-2c55-4000-90de-299660795719 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:18.917808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:18.918414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:18.918690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:18.918958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:18.920535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:18.969537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:19.013868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:19.015388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c5a5dd-d403-41d3-b555-4955bc3f519a service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:19.033845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 546/1105] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:20.087975 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f62e6b-2f9e-4728-9072-b24e467df552 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:20.088512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f62e6b-2f9e-4728-9072-b24e467df552 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:20.088512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f62e6b-2f9e-4728-9072-b24e467df552 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:20.089893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 560/1106] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:20.096006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49d3c4c-f16c-4caf-b2e1-4b63a990d225 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:20.096442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49d3c4c-f16c-4caf-b2e1-4b63a990d225 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:20.096535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49d3c4c-f16c-4caf-b2e1-4b63a990d225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:20.173967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d49d3c4c-f16c-4caf-b2e1-4b63a990d225 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:20.175159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d49d3c4c-f16c-4caf-b2e1-4b63a990d225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:20.268222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 547/1107] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:20.419041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f62e6b-2f9e-4728-9072-b24e467df552 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:20.420471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-89f62e6b-2f9e-4728-9072-b24e467df552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:20.469583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:20.566492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f62e6b-2f9e-4728-9072-b24e467df552 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:20.677951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:20.678479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:20.678750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:20.679020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:20.680596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:20.687417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:20.773462 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:20.775061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99caa3ce-8c5f-4b6f-b9c4-1cbb176e2a99 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:20.792642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 561/1108] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:21.845964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8a4afb-38a7-44e0-bb67-7f1e258e7ecd None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:21.846647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8a4afb-38a7-44e0-bb67-7f1e258e7ecd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:21.846647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8a4afb-38a7-44e0-bb67-7f1e258e7ecd None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:21.847852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 548/1109] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:21.854778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375c5394-b074-462f-bd00-e22bcc5b93b4 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:21.855235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375c5394-b074-462f-bd00-e22bcc5b93b4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:21.855340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375c5394-b074-462f-bd00-e22bcc5b93b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:21.930350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-375c5394-b074-462f-bd00-e22bcc5b93b4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:21.931550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-375c5394-b074-462f-bd00-e22bcc5b93b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:21.999099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:22.027720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 562/1110] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:22.183696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8a4afb-38a7-44e0-bb67-7f1e258e7ecd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:22.185248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8e8a4afb-38a7-44e0-bb67-7f1e258e7ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:22.202808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:22.329056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8a4afb-38a7-44e0-bb67-7f1e258e7ecd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:22.415533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:22.440493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:22.441129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:22.441404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:22.441672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:22.443251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:22.532346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:22.533937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a72ff-3772-47d4-9231-4298e92f5733 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:22.551368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 549/1111] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:23.603720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b974301-2659-4498-a4ba-7a638dc605a3 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:23.604261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b974301-2659-4498-a4ba-7a638dc605a3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:23.604261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b974301-2659-4498-a4ba-7a638dc605a3 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:23.605662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 563/1112] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:23.612030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2707b25-7671-4d95-8828-04e5283fb9c2 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:23.612765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2707b25-7671-4d95-8828-04e5283fb9c2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:23.612891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2707b25-7671-4d95-8828-04e5283fb9c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:23.688991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c2707b25-7671-4d95-8828-04e5283fb9c2 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:23.690207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c2707b25-7671-4d95-8828-04e5283fb9c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:23.731494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:23.786744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 550/1113] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:23.938703 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b974301-2659-4498-a4ba-7a638dc605a3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:23.940115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9b974301-2659-4498-a4ba-7a638dc605a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:24.082230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b974301-2659-4498-a4ba-7a638dc605a3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:24.138614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:24.198736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:24.199344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:24.199617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:24.199883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:24.201491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:24.289288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:24.290913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4abd85c-cdbb-4e21-b91f-5596d1c1bbce service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:24.308948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 564/1114] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:25.354433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1294a3a-c19b-45df-9833-446cffc0182d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:25.354999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1294a3a-c19b-45df-9833-446cffc0182d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:25.354999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1294a3a-c19b-45df-9833-446cffc0182d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:25.356316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 551/1115] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:23:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:25.362659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f0332c-5380-4259-acc3-40411d5fffdb None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:25.363090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f0332c-5380-4259-acc3-40411d5fffdb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:25.363215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f0332c-5380-4259-acc3-40411d5fffdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:25.439655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-49f0332c-5380-4259-acc3-40411d5fffdb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:25.440881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-49f0332c-5380-4259-acc3-40411d5fffdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:25.442855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:25.536068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 565/1116] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:23:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:25.684618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1294a3a-c19b-45df-9833-446cffc0182d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:25.686090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f1294a3a-c19b-45df-9833-446cffc0182d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:25.828777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1294a3a-c19b-45df-9833-446cffc0182d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:25.856213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:25.941790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:25.942398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:25.942672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:25.942939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:25.945073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:26.034458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:26.036024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c4c2a1-5d20-438b-a898-f5ea2807f355 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:26.053579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 552/1117] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:28.108733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c16cf0-c90e-45ff-8839-7836e600e7e8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:28.109279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c16cf0-c90e-45ff-8839-7836e600e7e8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:28.109279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c16cf0-c90e-45ff-8839-7836e600e7e8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:28.110561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 566/1118] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:28.116157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37def3cc-8d0a-4a69-bd71-163f5b753c42 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:28.116630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37def3cc-8d0a-4a69-bd71-163f5b753c42 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:28.116765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37def3cc-8d0a-4a69-bd71-163f5b753c42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:28.158174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:28.183306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-37def3cc-8d0a-4a69-bd71-163f5b753c42 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:28.184516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-37def3cc-8d0a-4a69-bd71-163f5b753c42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:28.279277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 553/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:28.564232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c16cf0-c90e-45ff-8839-7836e600e7e8 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:28.565854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a3c16cf0-c90e-45ff-8839-7836e600e7e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:28.668496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c16cf0-c90e-45ff-8839-7836e600e7e8 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:28.702055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:28.746158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:28.746756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:28.747030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:28.747297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:28.748901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:28.857629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:28.859033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bde733-c43f-4ea5-a738-1aec2818ef1e service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:28.877488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 567/1120] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:31.166543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d498e10-e6be-4302-9036-716bb3942aa1 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:31.166999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d498e10-e6be-4302-9036-716bb3942aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:31.166999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d498e10-e6be-4302-9036-716bb3942aa1 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:31.168368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 554/1121] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:31.173549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa3ab80-82ab-448a-bad4-3bfb6847a410 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:31.173992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa3ab80-82ab-448a-bad4-3bfb6847a410 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:31.174097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa3ab80-82ab-448a-bad4-3bfb6847a410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:31.242099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-caa3ab80-82ab-448a-bad4-3bfb6847a410 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:31.243428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-caa3ab80-82ab-448a-bad4-3bfb6847a410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:31.257745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:31.350542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 568/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:31.635033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d498e10-e6be-4302-9036-716bb3942aa1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:31.636604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9d498e10-e6be-4302-9036-716bb3942aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:31.737079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d498e10-e6be-4302-9036-716bb3942aa1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:31.750968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:31.811256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:31.811848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:31.812121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:31.812383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:31.813982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:31.922880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:31.924258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe02050-3b14-4938-879c-afd611f62fe7 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:31.942971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 555/1123] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:34.362090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5522fd8-2c9e-438f-9ffa-ab70cfb034f8 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:34.362896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5522fd8-2c9e-438f-9ffa-ab70cfb034f8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:34.362896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5522fd8-2c9e-438f-9ffa-ab70cfb034f8 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:34.364202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 569/1124] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:34.369563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a53fe6c-46b0-461e-81ec-5fbd3ea16158 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:34.369958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a53fe6c-46b0-461e-81ec-5fbd3ea16158 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:34.370060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a53fe6c-46b0-461e-81ec-5fbd3ea16158 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:34.417666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:34.435657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1a53fe6c-46b0-461e-81ec-5fbd3ea16158 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:34.436901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1a53fe6c-46b0-461e-81ec-5fbd3ea16158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:34.530871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 556/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:34.814556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5522fd8-2c9e-438f-9ffa-ab70cfb034f8 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:34.816137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d5522fd8-2c9e-438f-9ffa-ab70cfb034f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:34.920282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5522fd8-2c9e-438f-9ffa-ab70cfb034f8 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:34.960195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:34.995113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:34.995721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:34.995996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:34.996261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:34.997881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:35.104670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:35.106059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86e378c-e790-49bc-99c4-5943175465b9 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:35.124846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 570/1126] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:37.357556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d2bc8-e978-4a1e-b225-3e9756764236 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:37.358122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d2bc8-e978-4a1e-b225-3e9756764236 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:37.358122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d2bc8-e978-4a1e-b225-3e9756764236 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:37.359672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 557/1127] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:37.365437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f6341d-873f-4050-8eee-c3e214940a7a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:37.365870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f6341d-873f-4050-8eee-c3e214940a7a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:37.365976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f6341d-873f-4050-8eee-c3e214940a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:37.431389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-90f6341d-873f-4050-8eee-c3e214940a7a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:37.432507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-90f6341d-873f-4050-8eee-c3e214940a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:37.453299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:37.529796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 571/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:23:37.817092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190d2bc8-e978-4a1e-b225-3e9756764236 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:37.818623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-190d2bc8-e978-4a1e-b225-3e9756764236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:37.919139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190d2bc8-e978-4a1e-b225-3e9756764236 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:37.942775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:37.995742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:37.996337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:37.996607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:37.996978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:37.998546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:38.105319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:38.106663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a36a5c-6ac0-40e8-8a68-20701b7730cc service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:38.124354 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 558/1129] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:40.407644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618422fc-75c8-4868-ad70-b339bda0c995 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:40.408613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618422fc-75c8-4868-ad70-b339bda0c995 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:40.409168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618422fc-75c8-4868-ad70-b339bda0c995 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:40.410991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 572/1130] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:40.416862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672583c9-1d0b-4ac6-add8-a28e63af2c3c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:40.417274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672583c9-1d0b-4ac6-add8-a28e63af2c3c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:40.417408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672583c9-1d0b-4ac6-add8-a28e63af2c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:40.480912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:40.483906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-672583c9-1d0b-4ac6-add8-a28e63af2c3c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:40.485590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-672583c9-1d0b-4ac6-add8-a28e63af2c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:40.578458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 559/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:40.863189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618422fc-75c8-4868-ad70-b339bda0c995 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:40.864725 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-618422fc-75c8-4868-ad70-b339bda0c995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:40.965352 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618422fc-75c8-4868-ad70-b339bda0c995 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:41.014343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:41.038312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:41.038910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:41.039182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:41.039452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:41.041051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:41.148684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:41.150156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c63b1d-45e8-425e-9ef5-efeeeb5cc4cf service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:41.168322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 573/1132] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:43.368920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c490b0-1f4e-454f-8237-0bd485e08312 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:43.369441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c490b0-1f4e-454f-8237-0bd485e08312 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:43.369441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c490b0-1f4e-454f-8237-0bd485e08312 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:43.370819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 560/1133] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:43.376990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d7a960-e48f-4824-9148-68ce3b962017 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:43.377397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d7a960-e48f-4824-9148-68ce3b962017 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:43.377531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d7a960-e48f-4824-9148-68ce3b962017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:43.442585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-74d7a960-e48f-4824-9148-68ce3b962017 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:43.444317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-74d7a960-e48f-4824-9148-68ce3b962017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:43.473227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:43.540432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 574/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:43.823935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c490b0-1f4e-454f-8237-0bd485e08312 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:43.825489 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-69c490b0-1f4e-454f-8237-0bd485e08312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:43.927524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c490b0-1f4e-454f-8237-0bd485e08312 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:43.959854 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:44.003897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:44.004490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:44.004786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:44.005051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:44.007122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:44.114363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:44.115721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4208d1-2fe1-47da-b006-2a8441a1b96f service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:44.133954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 561/1135] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:46.322754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a205a70-b92a-4d47-878a-2f1b02ecaa60 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:46.323357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a205a70-b92a-4d47-878a-2f1b02ecaa60 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:46.323468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a205a70-b92a-4d47-878a-2f1b02ecaa60 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:46.324779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 575/1136] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:46.329987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71ba932-773a-40a1-a4b5-a2072cb1e62e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:46.330381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71ba932-773a-40a1-a4b5-a2072cb1e62e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:46.330517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71ba932-773a-40a1-a4b5-a2072cb1e62e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:46.396401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f71ba932-773a-40a1-a4b5-a2072cb1e62e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:46.397560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f71ba932-773a-40a1-a4b5-a2072cb1e62e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:46.405639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:46.493829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 562/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:46.774167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a205a70-b92a-4d47-878a-2f1b02ecaa60 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:46.775724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3a205a70-b92a-4d47-878a-2f1b02ecaa60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:46.878327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a205a70-b92a-4d47-878a-2f1b02ecaa60 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:46.933036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:46.951167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:46.951792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:46.952065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:46.952429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:46.953928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:47.061939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:47.063284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be74d627-69f3-449c-8718-deb32a33e905 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:47.081314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 576/1138] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:49.259233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0b3c00-d863-48c9-bb98-e195a618d58c None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:49.259796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0b3c00-d863-48c9-bb98-e195a618d58c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:49.259796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0b3c00-d863-48c9-bb98-e195a618d58c None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:49.261151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 563/1139] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:49.266355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e4034a-fd74-4315-855c-770176051d63 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:49.266779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e4034a-fd74-4315-855c-770176051d63 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:49.266885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e4034a-fd74-4315-855c-770176051d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:49.332144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-11e4034a-fd74-4315-855c-770176051d63 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:49.333362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-11e4034a-fd74-4315-855c-770176051d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:49.371921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:49.430007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 577/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:49.711550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0b3c00-d863-48c9-bb98-e195a618d58c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:49.713174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bc0b3c00-d863-48c9-bb98-e195a618d58c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:49.815572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0b3c00-d863-48c9-bb98-e195a618d58c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:49.853792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:49.889850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:49.890452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:49.890727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:49.890993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:49.892581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:50.001592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:50.002933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be586eea-54bd-47af-9cf3-2e6b1c3f9d47 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:50.020230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 564/1141] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:52.172632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed52cd4-f480-4d3f-86eb-da10a8bbf0fb None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:52.173179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed52cd4-f480-4d3f-86eb-da10a8bbf0fb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:52.173275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed52cd4-f480-4d3f-86eb-da10a8bbf0fb None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:52.174585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 578/1142] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:52.180387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013d19de-5cd8-4aaf-8e77-2e19fe49a01a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:52.180844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013d19de-5cd8-4aaf-8e77-2e19fe49a01a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:52.180951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013d19de-5cd8-4aaf-8e77-2e19fe49a01a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:52.253699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-013d19de-5cd8-4aaf-8e77-2e19fe49a01a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:52.254887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-013d19de-5cd8-4aaf-8e77-2e19fe49a01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:52.272341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:52.349071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 565/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:52.628931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed52cd4-f480-4d3f-86eb-da10a8bbf0fb None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:52.630468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-fed52cd4-f480-4d3f-86eb-da10a8bbf0fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:52.732155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed52cd4-f480-4d3f-86eb-da10a8bbf0fb None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:52.798133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:52.805280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:52.805906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:52.806180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:52.806448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:52.808140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:52.916511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:52.917985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a8d941-cf17-4871-92f8-771aaae4dbba service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:52.935454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 579/1144] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:23:55.077778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8961d602-d024-4bbd-abfa-1a50e110f163 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:55.078346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8961d602-d024-4bbd-abfa-1a50e110f163 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:55.078346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8961d602-d024-4bbd-abfa-1a50e110f163 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:55.079649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 566/1145] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:23:55.084886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97666ba6-b3a5-4a5f-a4ff-a94ebd09c586 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:55.085330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97666ba6-b3a5-4a5f-a4ff-a94ebd09c586 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:55.085458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97666ba6-b3a5-4a5f-a4ff-a94ebd09c586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:55.151384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-97666ba6-b3a5-4a5f-a4ff-a94ebd09c586 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:23:55.152587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-97666ba6-b3a5-4a5f-a4ff-a94ebd09c586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:55.200347 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:55.249603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 580/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:23:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:23:55.526766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8961d602-d024-4bbd-abfa-1a50e110f163 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:23:55.528284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8961d602-d024-4bbd-abfa-1a50e110f163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:55.628360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8961d602-d024-4bbd-abfa-1a50e110f163 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:23:55.675875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:23:55.701365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:23:55.701961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:23:55.702233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:23:55.702497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:23:55.704074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:23:55.813200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:23:55.814570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c90958c-e01b-42ba-a2d1-753a20934c92 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:23:55.831867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 567/1147] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:23:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:05.247940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=22 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:05.829652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=66 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:15.551868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e23cf9c-5bf0-4f94-9b1d-536b98e56470 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:15.552402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e23cf9c-5bf0-4f94-9b1d-536b98e56470 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:15.552402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e23cf9c-5bf0-4f94-9b1d-536b98e56470 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:15.553804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 581/1148] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:15.558996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6338e9b-23de-412e-93e0-90116a7e3e07 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:15.559445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6338e9b-23de-412e-93e0-90116a7e3e07 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:15.559556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6338e9b-23de-412e-93e0-90116a7e3e07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:15.624317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e6338e9b-23de-412e-93e0-90116a7e3e07 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:15.625536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e6338e9b-23de-412e-93e0-90116a7e3e07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:15.721794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 568/1149] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:16.019398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b591822-ab2a-48fa-8c25-fd37cb861ac5 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:16.019790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b591822-ab2a-48fa-8c25-fd37cb861ac5 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:16.020060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b591822-ab2a-48fa-8c25-fd37cb861ac5 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:16.021430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 582/1150] 10.222.0.23 () {54 vars in 962 bytes} [Wed Oct 12 02:24:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:16.027058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a82aa-6338-4747-8e4c-ed9da0dd2f25 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:16.027340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a82aa-6338-4747-8e4c-ed9da0dd2f25 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:16.027612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a82aa-6338-4747-8e4c-ed9da0dd2f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:16.103204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-889a82aa-6338-4747-8e4c-ed9da0dd2f25 None None] MFA Rules not processed for user `3d2788633742480981376ed8b171ac89`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:16.104348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-889a82aa-6338-4747-8e4c-ed9da0dd2f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:16.125493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:16.180680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 569/1151] 10.222.0.23 () {58 vars in 1059 bytes} [Wed Oct 12 02:24:16 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:16.188381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b591822-ab2a-48fa-8c25-fd37cb861ac5 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:16.189993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9b591822-ab2a-48fa-8c25-fd37cb861ac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:16.293283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b591822-ab2a-48fa-8c25-fd37cb861ac5 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:16.366526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:16.367052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:16.367327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:16.367664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:16.369343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:16.422443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:16.477793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:16.479232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d49f32-bbf2-4e1d-9e25-df957f5eb0b3 service glance] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:16.498944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 583/1152] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:24:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:18.724682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1306d39e-d559-4b06-8ba0-5faec717993d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:18.725337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1306d39e-d559-4b06-8ba0-5faec717993d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:18.725337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1306d39e-d559-4b06-8ba0-5faec717993d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:18.726693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 570/1153] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:18.732008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a213f7d-86af-4133-89ae-3ded56a412bb None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:18.732409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a213f7d-86af-4133-89ae-3ded56a412bb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:18.732604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a213f7d-86af-4133-89ae-3ded56a412bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:18.798213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3a213f7d-86af-4133-89ae-3ded56a412bb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:18.799396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3a213f7d-86af-4133-89ae-3ded56a412bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:18.894121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 584/1154] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:19.163360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1306d39e-d559-4b06-8ba0-5faec717993d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:19.164944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1306d39e-d559-4b06-8ba0-5faec717993d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:19.290625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1306d39e-d559-4b06-8ba0-5faec717993d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:19.339389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:19.383661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:19.384268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:19.384543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:19.384849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:19.385483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:19.386582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:24:19.386582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:24:19.386582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:24:19.387725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d50f18-a87b-429e-957a-c0743249a01a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:19.406346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 571/1155] 10.222.0.23 () {56 vars in 1108 bytes} [Wed Oct 12 02:24:19 2022] GET /identity/v3/projects => generated 2553 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:24:21.545477 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8985f83-1ca7-4cf8-99d9-ea974384bad7 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:21.546014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8985f83-1ca7-4cf8-99d9-ea974384bad7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:21.546014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8985f83-1ca7-4cf8-99d9-ea974384bad7 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:21.547332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 585/1156] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:21.552534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a95b7-9b5c-4dbe-ad93-e60f3c7fac80 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:21.553050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a95b7-9b5c-4dbe-ad93-e60f3c7fac80 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:21.553667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a95b7-9b5c-4dbe-ad93-e60f3c7fac80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:21.619591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b20a95b7-9b5c-4dbe-ad93-e60f3c7fac80 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:21.620809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b20a95b7-9b5c-4dbe-ad93-e60f3c7fac80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:21.714437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 572/1157] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:24:22.003062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8985f83-1ca7-4cf8-99d9-ea974384bad7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:22.004676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e8985f83-1ca7-4cf8-99d9-ea974384bad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:22.015163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:22.153216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8985f83-1ca7-4cf8-99d9-ea974384bad7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:22.235977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:22.268772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:22.269313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:22.269585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:22.269847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:22.271413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:22.360058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:22.361669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa74b76d-b760-4e56-afa7-554bde9792f8 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:22.379702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 586/1158] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:24:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:24.678764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fef2a7-8cef-4eff-ad68-f68d975d0c2d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:24.679364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fef2a7-8cef-4eff-ad68-f68d975d0c2d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:24.679364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fef2a7-8cef-4eff-ad68-f68d975d0c2d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:24.680642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 573/1159] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:24.685812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132a3fef-0fea-4a6c-b2c9-4b9cb8087e7d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:24.686219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132a3fef-0fea-4a6c-b2c9-4b9cb8087e7d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:24.686344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132a3fef-0fea-4a6c-b2c9-4b9cb8087e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:24.752400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-132a3fef-0fea-4a6c-b2c9-4b9cb8087e7d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:24.753638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-132a3fef-0fea-4a6c-b2c9-4b9cb8087e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:24.799683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:24.849698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 587/1160] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:25.135114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fef2a7-8cef-4eff-ad68-f68d975d0c2d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:25.136679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-33fef2a7-8cef-4eff-ad68-f68d975d0c2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:25.160537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:25.239551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fef2a7-8cef-4eff-ad68-f68d975d0c2d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:25.311484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:25.312083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:25.312357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:25.312626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:25.314252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:25.387586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:25.422094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:25.423521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1c0889-7135-4f56-8a54-a094a93efe7f service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:25.443340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 574/1161] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:24:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:27.596507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc2459b-f081-46d4-bb9d-5cac03d72894 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:27.597077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc2459b-f081-46d4-bb9d-5cac03d72894 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:27.597077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc2459b-f081-46d4-bb9d-5cac03d72894 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:27.598430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 588/1162] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:27.603827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97288f-4a33-44a3-8ddd-b765df647f08 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:27.604246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97288f-4a33-44a3-8ddd-b765df647f08 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:27.604398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97288f-4a33-44a3-8ddd-b765df647f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:27.670209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5b97288f-4a33-44a3-8ddd-b765df647f08 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:27.671404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5b97288f-4a33-44a3-8ddd-b765df647f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:27.769348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 575/1163] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:28.054903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc2459b-f081-46d4-bb9d-5cac03d72894 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:28.056430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2bc2459b-f081-46d4-bb9d-5cac03d72894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:28.161214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc2459b-f081-46d4-bb9d-5cac03d72894 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:28.234015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:28.235494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:28.236102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:28.236382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:28.236888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:28.238832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:28.347119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:28.348479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c077bfe-a0bc-498f-be48-7c875c323875 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:28.367907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 589/1164] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:24:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:30.604731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c41f6-24a8-4110-9ceb-d6f5499e3585 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:30.605288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c41f6-24a8-4110-9ceb-d6f5499e3585 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:30.605288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c41f6-24a8-4110-9ceb-d6f5499e3585 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:30.606659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 576/1165] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:30.612120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0960c686-f8f3-4848-a602-73f2c0c21253 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:30.612532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0960c686-f8f3-4848-a602-73f2c0c21253 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:30.612658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0960c686-f8f3-4848-a602-73f2c0c21253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:30.679646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0960c686-f8f3-4848-a602-73f2c0c21253 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:30.680883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0960c686-f8f3-4848-a602-73f2c0c21253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:30.732927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:30.777832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 590/1166] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:31.068952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07c41f6-24a8-4110-9ceb-d6f5499e3585 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:31.070535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e07c41f6-24a8-4110-9ceb-d6f5499e3585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:31.107441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:31.177994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07c41f6-24a8-4110-9ceb-d6f5499e3585 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:31.249994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:31.250597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:31.250873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:31.251141 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:31.252749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:31.330632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:31.359732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:31.361218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b37b6a-290e-4f9c-a9df-a9d6e008e687 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:31.379343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 577/1167] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:24:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:33.709608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f917dd62-f9b3-4cc0-abf0-a496cf92d409 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:33.710166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f917dd62-f9b3-4cc0-abf0-a496cf92d409 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:33.710166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f917dd62-f9b3-4cc0-abf0-a496cf92d409 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:33.711482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 591/1168] 10.222.0.23 () {54 vars in 927 bytes} [Wed Oct 12 02:24:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:24:33.716649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d73b57-50d6-4906-84d9-383fc317dd55 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:33.717111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d73b57-50d6-4906-84d9-383fc317dd55 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:33.717207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d73b57-50d6-4906-84d9-383fc317dd55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:33.787044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-50d73b57-50d6-4906-84d9-383fc317dd55 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:24:33.788835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-50d73b57-50d6-4906-84d9-383fc317dd55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:33.881669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 578/1169] 10.222.0.23 () {58 vars in 1024 bytes} [Wed Oct 12 02:24:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:24:34.170418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f917dd62-f9b3-4cc0-abf0-a496cf92d409 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:24:34.171942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f917dd62-f9b3-4cc0-abf0-a496cf92d409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:34.276473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f917dd62-f9b3-4cc0-abf0-a496cf92d409 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:24:34.348936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:24:34.349534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:24:34.350319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:24:34.350582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:24:34.352158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:24:34.360838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:34.461197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:24:34.462642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c652e2c2-6200-427e-ac43-e3262ca725e0 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:24:34.481185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 592/1170] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:24:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:24:43.880143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=82 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:24:44.478904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=50 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:52.982795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:52.983238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:52.983357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:53.058060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:25:53.059258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:53.153446 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 579/1171] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:25:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:25:53.161809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:53.162293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:53.162405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:53.239065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:25:53.240965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:53.337993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 593/1172] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:25:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:25:53.342600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:53.343798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:53.354890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec339c5-daea-41d2-aa61-bd3726e1c400 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:53.416169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:53.449981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:53.450584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:53.450859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:53.451130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:53.451765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:25:53.452064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:53.453093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:25:53.453093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:25:53.453093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:25:53.453973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:53.465136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 580/1173] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:25:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:25:53.474575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:53.483806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b28a4-b825-4bce-96af-b6fe4ab18081 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:53.544983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:53.578634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:53.579234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:53.579510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:53.579778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:53.580401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:53.581536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:25:53.581536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:25:53.581536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:25:53.582742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:53.637769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 594/1174] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:25:53 2022] POST /identity/v3/projects => generated 377 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:25:53.641966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:53.643466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:53.677284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddde7de-1435-4281-af02-1d1e126b211e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:53.771107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:53.771699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:53.771967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:53.772227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:53.772872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:53.773858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:25:53.773858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:25:53.773858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:25:53.775043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:53.815398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 581/1175] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:25:53 2022] POST /identity/v3/users => generated 289 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:25:53.820001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:53.821550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:53.883000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12380ad-1839-4ba3-8b55-17f4f957b419 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:53.965857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:53.976875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:53.977478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:53.977756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:53.978026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:53.978660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:53.980269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:53.993389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 595/1176] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:25:53 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:25:53.997643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:53.999147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:54.010801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d983df1b-e31b-4a91-9d46-ff0dfbf93ee2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:54.034969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:54.104012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd1e87d-7648-4f99-a935-9b872fd23725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:54.104542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd1e87d-7648-4f99-a935-9b872fd23725 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.104848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd1e87d-7648-4f99-a935-9b872fd23725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.105113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd1e87d-7648-4f99-a935-9b872fd23725 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.105737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd1e87d-7648-4f99-a935-9b872fd23725 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:54.107315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd1e87d-7648-4f99-a935-9b872fd23725 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:54.120350 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 582/1177] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:25:53 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:25:54.124411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:54.125989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:54.158899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e1a69-aa65-4c96-bc9b-eda3ac5621c5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:54.252614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bbea2c-9dbe-4e87-8d11-7d626ca438e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:54.253367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bbea2c-9dbe-4e87-8d11-7d626ca438e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.253640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bbea2c-9dbe-4e87-8d11-7d626ca438e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.253906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bbea2c-9dbe-4e87-8d11-7d626ca438e7 admin admin] PATH_INFO: `/v3/projects/a7115c1c110d4a9a9a51a287057141e3/users/78b491fa003a4aa4ade4443b2c1065ae/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.276156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bbea2c-9dbe-4e87-8d11-7d626ca438e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7115c1c110d4a9a9a51a287057141e3, user_id=78b491fa003a4aa4ade4443b2c1065ae, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:54.278835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bbea2c-9dbe-4e87-8d11-7d626ca438e7 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:54.302732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 596/1178] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:25:54 2022] PUT /identity/v3/projects/a7115c1c110d4a9a9a51a287057141e3/users/78b491fa003a4aa4ade4443b2c1065ae/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:25:54.313653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e11f13-23b1-4d37-8e27-58a6e1e24802 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.313938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e11f13-23b1-4d37-8e27-58a6e1e24802 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.314211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e11f13-23b1-4d37-8e27-58a6e1e24802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.370711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e5e11f13-23b1-4d37-8e27-58a6e1e24802 None None] MFA Rules not processed for user `78b491fa003a4aa4ade4443b2c1065ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:25:54.371848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e5e11f13-23b1-4d37-8e27-58a6e1e24802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:54.448819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 583/1179] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:25:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:25:54.462370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d339cb-0117-45cd-bf6b-c08ee592152a None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.462655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d339cb-0117-45cd-bf6b-c08ee592152a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.462928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d339cb-0117-45cd-bf6b-c08ee592152a None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.464254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 597/1180] 10.222.0.23 () {54 vars in 963 bytes} [Wed Oct 12 02:25:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:25:54.468965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f95cf2a-9e24-4ba5-9295-35434dcbd716 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.469244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f95cf2a-9e24-4ba5-9295-35434dcbd716 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.469542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f95cf2a-9e24-4ba5-9295-35434dcbd716 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.500367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:54.543733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-9f95cf2a-9e24-4ba5-9295-35434dcbd716 None None] MFA Rules not processed for user `575abe426dae4b868bb1c35537a9be35`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:25:54.544945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9f95cf2a-9e24-4ba5-9295-35434dcbd716 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:54.620081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 584/1181] 10.222.0.23 () {58 vars in 1060 bytes} [Wed Oct 12 02:25:54 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:25:54.628111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d339cb-0117-45cd-bf6b-c08ee592152a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:54.629590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-18d339cb-0117-45cd-bf6b-c08ee592152a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:54.679284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:54.733028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d339cb-0117-45cd-bf6b-c08ee592152a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:54.809317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:54.810034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.810308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.810573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.812144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:54.910167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:54.911531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:54.914986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:54.930675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 598/1182] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:25:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:25:54.975317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20875ac2-90a7-47d5-ab21-ba29dd087029 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:54.975697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20875ac2-90a7-47d5-ab21-ba29dd087029 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:54.975970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20875ac2-90a7-47d5-ab21-ba29dd087029 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:54.977249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 585/1183] 10.222.0.23 () {52 vars in 830 bytes} [Wed Oct 12 02:25:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:25:54.980841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:54.982240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:55.080964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8440e587-9add-4743-a57f-74858382b1ea service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:55.174105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:55.174662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:55.174934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:55.175198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] PATH_INFO: `/v3/users/78b491fa003a4aa4ade4443b2c1065ae` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:55.188273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=78b491fa003a4aa4ade4443b2c1065ae)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:55.189324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:25:55.189324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:25:55.189324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:25:55.190537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:55.200042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:55.255613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78b491fa003a4aa4ade4443b2c1065ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:55.278070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78b491fa003a4aa4ade4443b2c1065ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:55.278523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78b491fa003a4aa4ade4443b2c1065ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:55.288461 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78b491fa003a4aa4ade4443b2c1065ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:55.289008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78b491fa003a4aa4ade4443b2c1065ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:55.308584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 599/1184] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:25:54 2022] DELETE /identity/v3/users/78b491fa003a4aa4ade4443b2c1065ae => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:25:55.312947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:55.313436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:55.313561 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:55.371342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:25:55.372535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:55.446782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:55.459501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 586/1185] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:25:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:25:55.470407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:55.471618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:55.570259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdeb435-1665-42de-b679-5b89b30349ad admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:55.623815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:25:55.644856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:55.645465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:55.645741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:55.646011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:55.648093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:55.756943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:55.758299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882d3d01-c435-4a07-bf9c-f5d3e9eaa91a service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:55.776710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 600/1186] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:25:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:25:56.314688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:25:56.315924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:25:56.328380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eed3b83-943a-4b6d-85e0-7596b2f9ac5c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:25:56.423878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:25:56.424530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:25:56.424849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:25:56.425120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] PATH_INFO: `/v3/projects/a7115c1c110d4a9a9a51a287057141e3` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:25:56.436413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7115c1c110d4a9a9a51a287057141e3)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:25:56.437467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:25:56.437467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:25:56.437467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:25:56.438665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:25:56.495608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7115c1c110d4a9a9a51a287057141e3', 'request_id': 'req-957471b0-01ea-492e-8930-9d5acd712fa9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:56.501579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7115c1c110d4a9a9a51a287057141e3', 'request_id': 'req-957471b0-01ea-492e-8930-9d5acd712fa9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:56.511305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7115c1c110d4a9a9a51a287057141e3', 'request_id': 'req-957471b0-01ea-492e-8930-9d5acd712fa9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:56.540506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] The token cache is being invalidate because project a7115c1c110d4a9a9a51a287057141e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:25:56.540823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:56.541250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-957471b0-01ea-492e-8930-9d5acd712fa9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:25:56.542652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 587/1187] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:25:56 2022] DELETE /identity/v3/projects/a7115c1c110d4a9a9a51a287057141e3 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:26:05.774480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=62 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:26:06.538472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:27:22.893786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da3252d-84bb-4c13-9de6-a988e1aa10fe None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:22.894398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da3252d-84bb-4c13-9de6-a988e1aa10fe None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:22.894398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da3252d-84bb-4c13-9de6-a988e1aa10fe None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:22.895710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 601/1188] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:27:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:27:22.900500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c454daf3-4551-4a00-b0cb-28e848a37732 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:22.900981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c454daf3-4551-4a00-b0cb-28e848a37732 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:22.901091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c454daf3-4551-4a00-b0cb-28e848a37732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:22.982233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c454daf3-4551-4a00-b0cb-28e848a37732 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:22.983450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c454daf3-4551-4a00-b0cb-28e848a37732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:23.086805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 588/1189] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:27:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:23.242945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da3252d-84bb-4c13-9de6-a988e1aa10fe None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:23.245194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8da3252d-84bb-4c13-9de6-a988e1aa10fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:23.364374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da3252d-84bb-4c13-9de6-a988e1aa10fe None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:23.370673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccd14ed-aca0-47f6-9e83-49da24e52ee1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:23.371279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd14ed-aca0-47f6-9e83-49da24e52ee1 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:23.371558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd14ed-aca0-47f6-9e83-49da24e52ee1 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:23.371827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd14ed-aca0-47f6-9e83-49da24e52ee1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:23.379910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccd14ed-aca0-47f6-9e83-49da24e52ee1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:23.381555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccd14ed-aca0-47f6-9e83-49da24e52ee1 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:23.390938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 602/1190] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:24.436607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6051dad-6ec7-40ab-b26e-58851094f9ff None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:24.437168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6051dad-6ec7-40ab-b26e-58851094f9ff None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:24.437168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6051dad-6ec7-40ab-b26e-58851094f9ff None None] PATH_INFO: `/v3` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:24.438377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 589/1191] 10.222.0.16 () {54 vars in 925 bytes} [Wed Oct 12 02:27:24 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:27:24.444318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482f66f4-6131-45e7-9431-b841d95fbc7b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:24.445331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482f66f4-6131-45e7-9431-b841d95fbc7b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:24.445545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482f66f4-6131-45e7-9431-b841d95fbc7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:24.462691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9c57b2-a195-4cee-aa6c-d4d4027695ab None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:24.462969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9c57b2-a195-4cee-aa6c-d4d4027695ab None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:24.463240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9c57b2-a195-4cee-aa6c-d4d4027695ab None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:24.464550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 590/1192] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:27:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:27:24.470514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:24.470791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:24.471057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:24.490249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:24.492206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:24.499965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 591/1193] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:27:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:27:24.505480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-482f66f4-6131-45e7-9431-b841d95fbc7b None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:24.506653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-482f66f4-6131-45e7-9431-b841d95fbc7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:24.573827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:27:24.579032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 603/1194] 10.222.0.16 () {58 vars in 1015 bytes} [Wed Oct 12 02:27:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:24.651600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:24.664382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae3f90c-e230-4df2-87ca-ddabdbe7bf57 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:24.669765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407b2c01-ccd0-4473-b2c6-79d7a821cff3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:24.670361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407b2c01-ccd0-4473-b2c6-79d7a821cff3 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:24.670636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407b2c01-ccd0-4473-b2c6-79d7a821cff3 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:24.670905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407b2c01-ccd0-4473-b2c6-79d7a821cff3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:24.678873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407b2c01-ccd0-4473-b2c6-79d7a821cff3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:24.680436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407b2c01-ccd0-4473-b2c6-79d7a821cff3 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:24.689618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 592/1195] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:33.227874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367048a7-2911-47da-9422-14de2cab3e7b None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:33.228378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367048a7-2911-47da-9422-14de2cab3e7b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:33.228484 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367048a7-2911-47da-9422-14de2cab3e7b None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:33.229797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 604/1196] 10.222.0.23 () {54 vars in 961 bytes} [Wed Oct 12 02:27:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:33.234777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860167f7-2865-4f61-8f99-b1825ad58ad2 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:33.235213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860167f7-2865-4f61-8f99-b1825ad58ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:33.235324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860167f7-2865-4f61-8f99-b1825ad58ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:33.255939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-860167f7-2865-4f61-8f99-b1825ad58ad2 None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:33.257153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-860167f7-2865-4f61-8f99-b1825ad58ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:33.265538 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 593/1197] 10.222.0.23 () {58 vars in 1058 bytes} [Wed Oct 12 02:27:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:33.273029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367048a7-2911-47da-9422-14de2cab3e7b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:33.286604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367048a7-2911-47da-9422-14de2cab3e7b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:33.292119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401fb90-96d4-489c-bfa0-b8980bb74b12 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:33.292744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401fb90-96d4-489c-bfa0-b8980bb74b12 service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:33.293031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401fb90-96d4-489c-bfa0-b8980bb74b12 service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:33.293329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401fb90-96d4-489c-bfa0-b8980bb74b12 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:33.301717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401fb90-96d4-489c-bfa0-b8980bb74b12 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:33.303322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401fb90-96d4-489c-bfa0-b8980bb74b12 service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:33.313569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 605/1198] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:27:36.128336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051d87a7-5daa-483f-bffb-a4e95606413c None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:36.128827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051d87a7-5daa-483f-bffb-a4e95606413c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:36.128932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051d87a7-5daa-483f-bffb-a4e95606413c None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:36.130240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 594/1199] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:27:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:36.136441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36a12a7-77c0-4908-9b2a-8ce25216df37 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:36.136872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36a12a7-77c0-4908-9b2a-8ce25216df37 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:36.137033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36a12a7-77c0-4908-9b2a-8ce25216df37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:36.156602 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c36a12a7-77c0-4908-9b2a-8ce25216df37 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:36.157855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c36a12a7-77c0-4908-9b2a-8ce25216df37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:36.166326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 606/1200] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:27:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:36.323314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051d87a7-5daa-483f-bffb-a4e95606413c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:36.338016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051d87a7-5daa-483f-bffb-a4e95606413c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:36.343357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03ac4a-fa4f-4833-a83b-d78014a58a74 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:36.343953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03ac4a-fa4f-4833-a83b-d78014a58a74 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:36.344228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03ac4a-fa4f-4833-a83b-d78014a58a74 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:36.344494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03ac4a-fa4f-4833-a83b-d78014a58a74 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:36.353045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d03ac4a-fa4f-4833-a83b-d78014a58a74 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:36.354611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d03ac4a-fa4f-4833-a83b-d78014a58a74 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:36.363801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 595/1201] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:37.433107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4f720-9961-4c4c-9d6e-59672f13f58e None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:37.433637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4f720-9961-4c4c-9d6e-59672f13f58e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:37.433637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4f720-9961-4c4c-9d6e-59672f13f58e None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:37.434911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 607/1202] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 12 02:27:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:37.440938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d07cb4-c368-4a15-a13c-c81c716650d3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:37.441364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d07cb4-c368-4a15-a13c-c81c716650d3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:37.441489 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d07cb4-c368-4a15-a13c-c81c716650d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:37.460556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-65d07cb4-c368-4a15-a13c-c81c716650d3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:37.461783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-65d07cb4-c368-4a15-a13c-c81c716650d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:37.469491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 596/1203] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 12 02:27:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:37.623045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4f720-9961-4c4c-9d6e-59672f13f58e None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:37.637216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4f720-9961-4c4c-9d6e-59672f13f58e None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:37.642588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b67e66f-77fc-4615-bced-2afeae69eee7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:37.643174 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67e66f-77fc-4615-bced-2afeae69eee7 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:37.643451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67e66f-77fc-4615-bced-2afeae69eee7 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:37.643721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67e66f-77fc-4615-bced-2afeae69eee7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:37.651698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b67e66f-77fc-4615-bced-2afeae69eee7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:37.653325 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b67e66f-77fc-4615-bced-2afeae69eee7 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:37.662519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 608/1204] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:41.573953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651425b3-1b11-4acd-8d09-c1bca451bcc1 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:41.574520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651425b3-1b11-4acd-8d09-c1bca451bcc1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:41.574520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651425b3-1b11-4acd-8d09-c1bca451bcc1 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:41.575827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 597/1205] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:41.581110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63c7c5d-a99c-4729-8837-0854917fcf87 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:41.581535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63c7c5d-a99c-4729-8837-0854917fcf87 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:41.581642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63c7c5d-a99c-4729-8837-0854917fcf87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:41.601603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e63c7c5d-a99c-4729-8837-0854917fcf87 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:41.602778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e63c7c5d-a99c-4729-8837-0854917fcf87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:41.610337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 609/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:27:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:41.894456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651425b3-1b11-4acd-8d09-c1bca451bcc1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:41.896140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-651425b3-1b11-4acd-8d09-c1bca451bcc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:41.922457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:27:41.973675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651425b3-1b11-4acd-8d09-c1bca451bcc1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:41.977212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feba2f9f-7bb3-48bc-a3c4-aaa2ddc8c6b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:41.977806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba2f9f-7bb3-48bc-a3c4-aaa2ddc8c6b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:41.978079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba2f9f-7bb3-48bc-a3c4-aaa2ddc8c6b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:41.978340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba2f9f-7bb3-48bc-a3c4-aaa2ddc8c6b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:41.986964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feba2f9f-7bb3-48bc-a3c4-aaa2ddc8c6b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:41.988365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feba2f9f-7bb3-48bc-a3c4-aaa2ddc8c6b3 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:41.997672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 598/1207] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:44.128598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c7b24e-ee35-4de2-bcd9-cee5c436fd24 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:44.129059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c7b24e-ee35-4de2-bcd9-cee5c436fd24 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:44.129176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c7b24e-ee35-4de2-bcd9-cee5c436fd24 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:44.130521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 610/1208] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:27:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:44.135709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8fa2a5-15a1-4f34-8743-ad20d8146ae7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:44.136160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8fa2a5-15a1-4f34-8743-ad20d8146ae7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:44.136267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8fa2a5-15a1-4f34-8743-ad20d8146ae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:44.156070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-bf8fa2a5-15a1-4f34-8743-ad20d8146ae7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:44.157296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bf8fa2a5-15a1-4f34-8743-ad20d8146ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:44.165602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 599/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:27:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:44.453913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c7b24e-ee35-4de2-bcd9-cee5c436fd24 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:44.455639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-55c7b24e-ee35-4de2-bcd9-cee5c436fd24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:44.468584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c7b24e-ee35-4de2-bcd9-cee5c436fd24 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:44.472103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408a4a3c-d79f-423a-917d-018914204cba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:44.472719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a4a3c-d79f-423a-917d-018914204cba service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:44.473002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a4a3c-d79f-423a-917d-018914204cba service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:44.473272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a4a3c-d79f-423a-917d-018914204cba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:44.482586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408a4a3c-d79f-423a-917d-018914204cba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:44.484026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408a4a3c-d79f-423a-917d-018914204cba service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:44.493864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 611/1210] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:48.059124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea55477-0ae1-4a05-9ead-a811a826a9c3 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:48.059695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea55477-0ae1-4a05-9ead-a811a826a9c3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:48.059695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea55477-0ae1-4a05-9ead-a811a826a9c3 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:48.061052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 600/1211] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:27:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:48.066219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7b9d3-7c57-468e-8df3-7209130da08b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:48.066621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7b9d3-7c57-468e-8df3-7209130da08b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:48.066761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7b9d3-7c57-468e-8df3-7209130da08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:48.086178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-26a7b9d3-7c57-468e-8df3-7209130da08b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:48.087383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-26a7b9d3-7c57-468e-8df3-7209130da08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:48.095173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 612/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:27:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:48.397848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea55477-0ae1-4a05-9ead-a811a826a9c3 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:48.399522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5ea55477-0ae1-4a05-9ead-a811a826a9c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:48.476434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea55477-0ae1-4a05-9ead-a811a826a9c3 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:48.480276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc530f4-3081-4707-ab3f-e0ce061ca2ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:48.480902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc530f4-3081-4707-ab3f-e0ce061ca2ad service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:48.481178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc530f4-3081-4707-ab3f-e0ce061ca2ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:48.481448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc530f4-3081-4707-ab3f-e0ce061ca2ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:48.490162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc530f4-3081-4707-ab3f-e0ce061ca2ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:48.492340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc530f4-3081-4707-ab3f-e0ce061ca2ad service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:48.501625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 601/1213] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:27:50.622683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3545f1c-67f9-4270-bb3c-561946cc2356 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:50.623223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3545f1c-67f9-4270-bb3c-561946cc2356 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:50.623223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3545f1c-67f9-4270-bb3c-561946cc2356 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:50.624555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 613/1214] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 12 02:27:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:27:50.629719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdb6999-3243-4234-a9a6-6907d5d94385 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:50.630153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdb6999-3243-4234-a9a6-6907d5d94385 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:50.630262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdb6999-3243-4234-a9a6-6907d5d94385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:50.650277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-bfdb6999-3243-4234-a9a6-6907d5d94385 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:27:50.651472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bfdb6999-3243-4234-a9a6-6907d5d94385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:50.659166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 602/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 12 02:27:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:27:50.966092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3545f1c-67f9-4270-bb3c-561946cc2356 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:27:50.967761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d3545f1c-67f9-4270-bb3c-561946cc2356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:27:50.980736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3545f1c-67f9-4270-bb3c-561946cc2356 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:27:50.984158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d2099-aae3-4d5c-9f53-a97c43b2b68b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:27:50.984775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d2099-aae3-4d5c-9f53-a97c43b2b68b service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:27:50.985050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d2099-aae3-4d5c-9f53-a97c43b2b68b service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:27:50.985315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d2099-aae3-4d5c-9f53-a97c43b2b68b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:27:50.994275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d2099-aae3-4d5c-9f53-a97c43b2b68b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:27:50.995693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d2099-aae3-4d5c-9f53-a97c43b2b68b service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:27:51.004796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 614/1216] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:27:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:28:00.644823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=66 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:28:01.000854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=34 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:14.472821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.473470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.473470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.500038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.501694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.516847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 603/1217] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:14.532619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.533233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.533233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.564848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.566177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.578380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 615/1218] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:14.583575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:14.596070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a3430e-c843-44d4-b41b-2f3eda14290a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:14.605229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:14.606187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.606685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.607183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.608045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:14.608568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:14.610437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:14.628456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 604/1219] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:14.637119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:14.653822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca64165-26e4-4c08-8f37-64532f130e17 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:14.661607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb4fc00-959b-4bd7-a996-1b16067da007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:14.662251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb4fc00-959b-4bd7-a996-1b16067da007 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.663099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb4fc00-959b-4bd7-a996-1b16067da007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.663372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb4fc00-959b-4bd7-a996-1b16067da007 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.664028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb4fc00-959b-4bd7-a996-1b16067da007 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:14.666031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb4fc00-959b-4bd7-a996-1b16067da007 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:14.723706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 616/1220] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/projects => generated 371 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:14.728337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:14.745069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0cd49b-2bda-4ec7-9c55-24417ef1676b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:14.756788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2d8a60-c0e4-45e9-8eda-f6508b188385 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.757137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2d8a60-c0e4-45e9-8eda-f6508b188385 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.757354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2d8a60-c0e4-45e9-8eda-f6508b188385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.792415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-cf2d8a60-c0e4-45e9-8eda-f6508b188385 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.792415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cf2d8a60-c0e4-45e9-8eda-f6508b188385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.838116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2fc332-d0c2-4b21-af9f-017da5b7d2ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:14.839259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fc332-d0c2-4b21-af9f-017da5b7d2ff admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.839689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fc332-d0c2-4b21-af9f-017da5b7d2ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.840123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fc332-d0c2-4b21-af9f-017da5b7d2ff admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.840963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fc332-d0c2-4b21-af9f-017da5b7d2ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:14.843054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fc332-d0c2-4b21-af9f-017da5b7d2ff admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:14.875186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 617/1221] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:14.878495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 605/1222] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:14.881671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8e3c48-9818-4338-936e-1408ea2583c5 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.882355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.882793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.883219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.886309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8e3c48-9818-4338-936e-1408ea2583c5 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.886811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8e3c48-9818-4338-936e-1408ea2583c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.921947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.923445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.924884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-bb8e3c48-9818-4338-936e-1408ea2583c5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.926150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-bb8e3c48-9818-4338-936e-1408ea2583c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.931934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 606/1223] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:14.932838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:14.935188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 618/1224] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:14.935629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e894eca-376c-439b-bc51-020f01d12066 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.937938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e894eca-376c-439b-bc51-020f01d12066 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.937938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e894eca-376c-439b-bc51-020f01d12066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.945304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb8489d-19d9-4cab-a42a-020be4c86e0a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:14.950898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c03acd-7468-4ddf-8a78-3a43eeb02802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:14.951519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c03acd-7468-4ddf-8a78-3a43eeb02802 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.951936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c03acd-7468-4ddf-8a78-3a43eeb02802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.952113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c03acd-7468-4ddf-8a78-3a43eeb02802 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.952808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c03acd-7468-4ddf-8a78-3a43eeb02802 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:14.954448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c03acd-7468-4ddf-8a78-3a43eeb02802 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:14.959856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-9e894eca-376c-439b-bc51-020f01d12066 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.961632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9e894eca-376c-439b-bc51-020f01d12066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.966340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 607/1225] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:14.970262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 619/1226] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:14.970415 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d67668e-e8a5-43ac-9b51-96a1fa4007f7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.971426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d67668e-e8a5-43ac-9b51-96a1fa4007f7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.971426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d67668e-e8a5-43ac-9b51-96a1fa4007f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.971633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65efe84e-51b4-4544-8d70-c4f4e1353afe None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:14.971633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65efe84e-51b4-4544-8d70-c4f4e1353afe None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:14.971896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65efe84e-51b4-4544-8d70-c4f4e1353afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:14.990948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6d67668e-e8a5-43ac-9b51-96a1fa4007f7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.991143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-65efe84e-51b4-4544-8d70-c4f4e1353afe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:14.992068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6d67668e-e8a5-43ac-9b51-96a1fa4007f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.992239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-65efe84e-51b4-4544-8d70-c4f4e1353afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:14.999871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 608/1227] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.000007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 620/1228] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.001133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5054c-e72b-488b-9134-336a7b94d733 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.001273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.001440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5054c-e72b-488b-9134-336a7b94d733 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.001555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.001651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5054c-e72b-488b-9134-336a7b94d733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.001735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.022492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.022740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4fa5054c-e72b-488b-9134-336a7b94d733 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.023713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.023845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4fa5054c-e72b-488b-9134-336a7b94d733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.031931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 609/1229] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.032062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 621/1230] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.033147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.033419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.033686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.034410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.044852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1ec0a6-05ee-4cb7-b524-9236b484d0f2 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.049680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194857f5-888e-4c6b-ae6e-cc2dacc0adbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.050286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194857f5-888e-4c6b-ae6e-cc2dacc0adbb admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.050550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194857f5-888e-4c6b-ae6e-cc2dacc0adbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.050817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194857f5-888e-4c6b-ae6e-cc2dacc0adbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.051452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194857f5-888e-4c6b-ae6e-cc2dacc0adbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.053128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194857f5-888e-4c6b-ae6e-cc2dacc0adbb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.056028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.057258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.063386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 622/1231] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:15.064523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e9dc7-db85-422e-8dc6-82d4fe274cd3 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.064836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e9dc7-db85-422e-8dc6-82d4fe274cd3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.065109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e9dc7-db85-422e-8dc6-82d4fe274cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.065436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 610/1232] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.066219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.079771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2a4023-232e-4272-ac00-fd7001b1e783 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.084992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.085338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-866e9dc7-db85-422e-8dc6-82d4fe274cd3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.085581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.085874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.086131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.087025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-866e9dc7-db85-422e-8dc6-82d4fe274cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.087124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.087124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.088632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0646c4fb-cf6f-487e-8102-92d6e876bcea admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.095396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 623/1233] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.096579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.096892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.097161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.104217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 611/1234] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.105837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327021aa-6e29-4cda-beb4-148217a7e1f5 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.106123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327021aa-6e29-4cda-beb4-148217a7e1f5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.106400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327021aa-6e29-4cda-beb4-148217a7e1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.117943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.118742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.126514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 624/1235] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.127286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.127903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-327021aa-6e29-4cda-beb4-148217a7e1f5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.129119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-327021aa-6e29-4cda-beb4-148217a7e1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.137320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83638678-ba4e-486c-b417-3eb2829ed6d2 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.137886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 612/1236] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.137886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.137886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.137886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.141995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.142582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.142856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.143122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.143761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.144058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.145094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:29:15.145094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:29:15.145094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:29:15.145974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.158750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 625/1237] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.159365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.159476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.160581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.168541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 613/1238] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.169007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ff0646-be49-4a47-af79-bae2304e7fba admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.170090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.173561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.174270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.174544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.174958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] PATH_INFO: `/v3/projects/ccda4a4b3d95486caaeb607896fdaced/users/077facfd33854415ad33ee217c4f7121/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.179908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122fc997-7ae7-4e82-93f8-676cd0649c29 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.184638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.185393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.185727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.186006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.186652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.186951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.188542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89766815-a11b-4aa8-92af-2482adb5f9aa admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.189655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccda4a4b3d95486caaeb607896fdaced, user_id=077facfd33854415ad33ee217c4f7121, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.193017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.201112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 614/1239] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.202858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.203195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.203516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.210360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 626/1240] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:15 2022] PUT /identity/v3/projects/ccda4a4b3d95486caaeb607896fdaced/users/077facfd33854415ad33ee217c4f7121/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:15.213523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.223913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.225173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.253149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:15.275062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:15.278556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5d628-2bce-4f06-92f7-e591b289d889 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.283331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.283942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.284220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.284489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.285152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.287082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.288074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 615/1241] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.288927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.297951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2dc289-2610-4ef5-850c-76cafa4af2c6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.302524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.303661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.303937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.304331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.304903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.305204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.306796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.316175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 627/1242] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.326887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 616/1243] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.327052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.327158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.336349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd5f3b1-fbb8-4f5d-b137-923f04a31819 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.388729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804e203b-92f7-4e3a-9c19-8af02a15fedf admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.392893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.393319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.393523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.393783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.393901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.394047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.394180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.394445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.394680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.395083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.395381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.395687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:29:15.395687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:29:15.395687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:29:15.396985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.397099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.410473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 628/1244] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.411280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.421096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81be3be0-7e08-45e1-b0ec-9612b5c7074f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.426424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.427013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.427290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.427546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.428679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.429221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.431581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 617/1245] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 389 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.431797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed26cec-4de6-4284-9c06-faece19d38fa admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.432402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.447217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 629/1246] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.448451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.448763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.449034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.476440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] MFA Rules not processed for user `077facfd33854415ad33ee217c4f7121`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:15.477688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:15.495413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa2e303-6a54-41f1-89ec-7a8ac2ff2041 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.500031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.500641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.501021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.501295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.501934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.503895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.531472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 618/1247] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 385 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.532443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.544075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 630/1248] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:15.545628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.556668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98d5c0d-abff-4a91-8d4c-73c7bcced209 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.585383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:15.594860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:15.607372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2247c2f3-dac4-400a-af94-ac479dd093ce admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.612687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.613370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.613698 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.613974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.614645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:15.614949 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.617261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.617506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.617861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.618141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.618406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.619027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.620030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:29:15.620030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:29:15.620030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:29:15.621277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.629344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 619/1249] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:15.630237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.642778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 631/1250] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.643784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.645940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a144ab2e-6c48-479e-864f-8ebef9124dc0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.650639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.651244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.651523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.651790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.652416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.653247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2060591-6c5d-47f2-a436-f83dc97192ff admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.654340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.657714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.658282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.658556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.658849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.659740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.661653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.687105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 620/1251] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 391 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.687105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.690735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 632/1252] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 379 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.691555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.746400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268de1cc-1a98-44c9-a30f-09105e32d32d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.751068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.751672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.751950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.752217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.752874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.754183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c382f0-e707-4846-9d1a-a8aa460fcd95 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.754789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.758848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.759466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.759747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.760015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.760630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.762664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.776365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 621/1253] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/users => generated 295 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.777506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.788235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266277e-d9ec-4e17-a7cd-40edd35bc7d9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.795884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.796069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 633/1254] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 373 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.796170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.796250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.796333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.796333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.796333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.796575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.817025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 622/1255] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/users => generated 293 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.817800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.829773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3074154-f5c6-4cd9-9304-48f6940e9eb1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.860535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecf0077-ea21-45df-9760-6f2280859bfb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.867084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.867700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.867980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.868256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.868913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.870850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.890624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.891379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.891761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.892037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.892905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.894813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:15.898802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 634/1256] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 371 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.899815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.909789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:15.925893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 623/1257] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/projects => generated 387 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:15.927134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:15.947632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:15.965543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a5d32a-35fb-42d6-969f-b4abe92c1b4c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.989150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1af1bb-548f-49a1-9ce0-5ad5c939e737 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:15.994554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:15.995170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:15.995442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:15.995719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:15.996347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:15.998322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.023034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 624/1258] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:15 2022] POST /identity/v3/users => generated 296 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:16.023034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.025164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.025779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.026056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.026326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.026963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.028579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.030546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1de18db-e098-45e1-8f82-45d430a74da0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.035277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.035999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.036359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.036555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.037202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.041760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 635/1259] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:15 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.041920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.042003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.049001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f07777c-2c24-46e0-b542-096998d2a6fe admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.055520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.056576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.056977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.057244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.057870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.059445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.063923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 625/1260] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/users => generated 290 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:16.063923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.072223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7303ebe-5fe9-41a3-bad1-d6a001a32d50 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.076024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 636/1261] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.076024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.080525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.081155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.081425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.081689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.082304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.083404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84abd994-a508-4074-bc54-24b10edbbceb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.084465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.088011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.088580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.088894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.089166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.089781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.091373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.102759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 637/1262] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.103568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.107542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 626/1263] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/users => generated 287 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:16.108501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.114480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85384f0-6e42-406b-92eb-ab271dc70b1b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.120534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.121200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.121474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.121696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9743acd-e746-41f7-88ca-df12743b2491 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.121804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.122351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.124304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.126386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.126976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.127245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.127505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.128149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.130055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.148624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 638/1264] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/users => generated 292 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:16.149716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.152854 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 627/1265] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/users => generated 294 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:16.153670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.159773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195bbc8f-b889-4d45-b93b-bf5a1dbcf53f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.163191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884c06b5-a46e-436a-95b0-eb1300eb18e3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.164258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.164871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.165142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.165406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.166061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.167675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.167804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.168373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.168649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.168951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.169582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.171140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.179154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 639/1266] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.180013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.184648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 628/1267] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.184648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.191664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93929cd-d361-47d0-b283-3120af9cbcd0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.194200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a429c635-3425-44ab-8144-5dd0005daef4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.196259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.196878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.197156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.197427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.198042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.198854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.199436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.199608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.199704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.199960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.200579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.202357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.210334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 629/1268] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.211124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.213078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 640/1269] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.216733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.220800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634e716-3d03-4b0f-b25d-bcd8215f5bdd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.225867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.226119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7825e76a-7379-4cec-a3e4-4b2f4095c981 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.226451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.226760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.226995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.227612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.229216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.231502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.232077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.232350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.232794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.233432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.235465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.238975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 630/1270] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.239833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.250955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 641/1271] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.250955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.251220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431013d-41e9-417a-8a18-05a4347b61b0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.255136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.255733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.256058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.256332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.256996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.258607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbdf7d3-cf87-4ece-9aa7-0b53a97dcaab admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.258722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.263076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.263648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.263931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.264182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.264824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.266706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.268538 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 631/1272] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.269361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.280850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 642/1273] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.280850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.281095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec86845a-31f0-44c5-af6e-40b65d9720ce admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.285226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.285816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.286091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.286358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.286973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.288307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b9367-9cfa-4c60-a4a3-ff7c409f2255 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.288596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.293227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.293844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.294116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.294381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] PATH_INFO: `/v3/projects/2d2ef7ec625d4fd79045b3d5001dc790/users/f428a9f1b0834b89b4a87e48469d224a/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.301877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 632/1274] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.301877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.309305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f796a35-ed90-462f-9ec0-5deb020779d0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.312133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d2ef7ec625d4fd79045b3d5001dc790, user_id=f428a9f1b0834b89b4a87e48469d224a, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.313943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.314586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.314828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.314934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.315126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] PATH_INFO: `/v3/projects/7f6eb3e59c464c749edb69e8370f54e5/users/43f16ca13925411fb76a330b6c95395d/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.327897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 643/1275] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/2d2ef7ec625d4fd79045b3d5001dc790/users/f428a9f1b0834b89b4a87e48469d224a/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.328685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.333655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:16.341722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f6eb3e59c464c749edb69e8370f54e5, user_id=43f16ca13925411fb76a330b6c95395d, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.344638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.358422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 633/1276] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/7f6eb3e59c464c749edb69e8370f54e5/users/43f16ca13925411fb76a330b6c95395d/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.362257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.397100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c4682b-ef84-4a29-8fa9-f8cabca3a1e5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.402643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=87,keystone:INSERT=13 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:16.424077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d5360-388a-464f-9f9e-2b756ec70add admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.429199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.429879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.430154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.430422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] PATH_INFO: `/v3/projects/76e921b067f04677b76c245b5f93e112/users/1fceec891ef24ba3a39da72c90fb733a/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.445123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] RBAC: Authorizing `identity:create_grant(project_id=76e921b067f04677b76c245b5f93e112, user_id=1fceec891ef24ba3a39da72c90fb733a, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.448054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.454241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.454849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.455123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.455438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.456217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.457883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.460918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 634/1277] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/76e921b067f04677b76c245b5f93e112/users/1fceec891ef24ba3a39da72c90fb733a/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.464751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.468137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 644/1278] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.469001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.478937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cb862c-7f11-4d21-ae55-1f2bf56718f5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.525265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495db753-9be9-4bd8-9dae-555c8070080e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.530018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.530620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.530948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.531164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.531775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.533419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.533908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.534581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.534851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.535143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] PATH_INFO: `/v3/projects/a0fc9e9f616e45eab32cc99f24af1bee/users/89edce365b9f41648655523e9d7c68d5/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.543703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 635/1279] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.547757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.550559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0fc9e9f616e45eab32cc99f24af1bee, user_id=89edce365b9f41648655523e9d7c68d5, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.553978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.559175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c198b64d-9227-4a87-ab89-a1f0b9223a47 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.564810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b400903c-66b6-4b75-b1d6-df08ce3fa876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.565399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b400903c-66b6-4b75-b1d6-df08ce3fa876 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.565667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b400903c-66b6-4b75-b1d6-df08ce3fa876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.565931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b400903c-66b6-4b75-b1d6-df08ce3fa876 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.566548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b400903c-66b6-4b75-b1d6-df08ce3fa876 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.567453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 645/1280] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/a0fc9e9f616e45eab32cc99f24af1bee/users/89edce365b9f41648655523e9d7c68d5/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.568145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.571246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b400903c-66b6-4b75-b1d6-df08ce3fa876 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.581418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 636/1281] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.582481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8734c897-b19f-4f26-a974-32b69fbf9c33 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.582758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8734c897-b19f-4f26-a974-32b69fbf9c33 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.583028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8734c897-b19f-4f26-a974-32b69fbf9c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.605029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:16.610699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8734c897-b19f-4f26-a974-32b69fbf9c33 None None] MFA Rules not processed for user `f428a9f1b0834b89b4a87e48469d224a`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:16.612063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8734c897-b19f-4f26-a974-32b69fbf9c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:16.634962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16980830-d66c-41ce-8895-25dea263977c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.641275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b93225c-1cd1-49e6-ba59-1d7910bd74e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.642194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b93225c-1cd1-49e6-ba59-1d7910bd74e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.642463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b93225c-1cd1-49e6-ba59-1d7910bd74e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.642721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b93225c-1cd1-49e6-ba59-1d7910bd74e4 admin admin] PATH_INFO: `/v3/projects/bd41b04d606949edbc9a3a2f88f991b2/users/bcd7f1121dab4e8db4c747c51ed49240/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.657273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b93225c-1cd1-49e6-ba59-1d7910bd74e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd41b04d606949edbc9a3a2f88f991b2, user_id=bcd7f1121dab4e8db4c747c51ed49240, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.659926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b93225c-1cd1-49e6-ba59-1d7910bd74e4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.672869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 646/1282] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/bd41b04d606949edbc9a3a2f88f991b2/users/bcd7f1121dab4e8db4c747c51ed49240/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.674101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.674385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.674641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.679398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 637/1283] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:16.682115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.682500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.682903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.694058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:16.695268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:16.702701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:16.703888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:16.738817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:16.757578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 647/1284] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:16.758402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.767454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd4b56-9550-4da0-b497-0e17bf173f75 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.772819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.773112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 638/1285] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:16.774042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.776630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.776908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.777190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] PATH_INFO: `/v3/projects/60e8d0fb43eb419ba1c5eb830cc89ee3/users/47f30e76f6894e5f9c06f71adde020e6/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.784419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630878fc-dc4a-4c47-ac94-f91cc03a272b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.790339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.790975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.791253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.791523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] PATH_INFO: `/v3/projects/90817c5c259e47f2aaebfe96f44cd58f/users/5030d50b94a34b9d8958e080cf1c9c7c/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.792904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60e8d0fb43eb419ba1c5eb830cc89ee3, user_id=47f30e76f6894e5f9c06f71adde020e6, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.795580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.806219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] RBAC: Authorizing `identity:create_grant(project_id=90817c5c259e47f2aaebfe96f44cd58f, user_id=5030d50b94a34b9d8958e080cf1c9c7c, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.808310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 648/1286] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/60e8d0fb43eb419ba1c5eb830cc89ee3/users/47f30e76f6894e5f9c06f71adde020e6/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.808897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.809270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.822170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 639/1287] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/90817c5c259e47f2aaebfe96f44cd58f/users/5030d50b94a34b9d8958e080cf1c9c7c/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.823701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.872977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fab343-11e2-461d-8626-dd904c7419b1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.894394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86529ea7-c1d0-4ab2-8f02-8945edc43cff admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:16.899945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9732d12b-c40c-4f4b-af94-463de702c7f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.900600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9732d12b-c40c-4f4b-af94-463de702c7f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.900903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9732d12b-c40c-4f4b-af94-463de702c7f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.901170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9732d12b-c40c-4f4b-af94-463de702c7f2 admin admin] PATH_INFO: `/v3/projects/fe0a12a6b1564071955fe9c37fd5534c/users/93727a50124b4ac995770d38ef23e43f/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.905568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:16.916775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9732d12b-c40c-4f4b-af94-463de702c7f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe0a12a6b1564071955fe9c37fd5534c, user_id=93727a50124b4ac995770d38ef23e43f, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.919448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9732d12b-c40c-4f4b-af94-463de702c7f2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.931194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a891e2d6-003d-42dd-95e7-935a04157d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:16.931838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a891e2d6-003d-42dd-95e7-935a04157d2b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.932049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 640/1288] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:16 2022] PUT /identity/v3/projects/fe0a12a6b1564071955fe9c37fd5534c/users/93727a50124b4ac995770d38ef23e43f/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:16.932180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a891e2d6-003d-42dd-95e7-935a04157d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.932416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a891e2d6-003d-42dd-95e7-935a04157d2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.933214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a891e2d6-003d-42dd-95e7-935a04157d2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:16.933542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.933849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.934150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.935037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a891e2d6-003d-42dd-95e7-935a04157d2b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:16.946281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 649/1289] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:16.947605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d08d51f-7b30-4d13-82ac-197a80104544 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:16.947901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d08d51f-7b30-4d13-82ac-197a80104544 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:16.948166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d08d51f-7b30-4d13-82ac-197a80104544 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:16.949489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 650/1290] 10.222.0.23 () {54 vars in 961 bytes} [Wed Oct 12 02:29:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:16.950240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d08d51f-7b30-4d13-82ac-197a80104544 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:16.958799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:16.960525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:17.009054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d08d51f-7b30-4d13-82ac-197a80104544 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.012797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bea4a8-b548-4f30-83f6-2a4c344e33cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.013488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bea4a8-b548-4f30-83f6-2a4c344e33cd service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.013847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bea4a8-b548-4f30-83f6-2a4c344e33cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.014143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bea4a8-b548-4f30-83f6-2a4c344e33cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.020341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 641/1291] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:17.024245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bea4a8-b548-4f30-83f6-2a4c344e33cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.025056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.025720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bea4a8-b548-4f30-83f6-2a4c344e33cd service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.034622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ca1cb-daf0-415e-a4a2-5873195352ac None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.035672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 651/1292] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:17.037042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d49b7c-4ff6-4771-85b7-db01d85d28b4 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.037394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d49b7c-4ff6-4771-85b7-db01d85d28b4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.037675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d49b7c-4ff6-4771-85b7-db01d85d28b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.038453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f5ae37-fb59-4541-8d04-5ae7569ea87d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.039054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f5ae37-fb59-4541-8d04-5ae7569ea87d service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.039329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f5ae37-fb59-4541-8d04-5ae7569ea87d service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.039598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f5ae37-fb59-4541-8d04-5ae7569ea87d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.049619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f5ae37-fb59-4541-8d04-5ae7569ea87d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.050642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f5ae37-fb59-4541-8d04-5ae7569ea87d service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.057325 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d2d49b7c-4ff6-4771-85b7-db01d85d28b4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:17.058556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d2d49b7c-4ff6-4771-85b7-db01d85d28b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:17.060952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 642/1293] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:17.062172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15c7de-65b3-438e-bf0c-fa94021cdbb1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.062820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15c7de-65b3-438e-bf0c-fa94021cdbb1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.063098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15c7de-65b3-438e-bf0c-fa94021cdbb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.071349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 652/1294] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:17.071349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.071349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.071349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.102054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] MFA Rules not processed for user `93727a50124b4ac995770d38ef23e43f`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:17.103310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:17.106857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2b15c7de-65b3-438e-bf0c-fa94021cdbb1 None None] MFA Rules not processed for user `5030d50b94a34b9d8958e080cf1c9c7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:17.109019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2b15c7de-65b3-438e-bf0c-fa94021cdbb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:17.122655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:17.180399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 653/1295] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:17.181110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.190236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 643/1296] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:17.190236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.190236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.190236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.194913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e19d5-a44a-463c-975b-74da68205a28 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.201906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.201906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.202498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.202619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] PATH_INFO: `/v3/projects/a0fc9e9f616e45eab32cc99f24af1bee/users/89edce365b9f41648655523e9d7c68d5/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.204831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0fc9e9f616e45eab32cc99f24af1bee, user_id=89edce365b9f41648655523e9d7c68d5, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.207631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.219031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 654/1297] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:17 2022] PUT /identity/v3/projects/a0fc9e9f616e45eab32cc99f24af1bee/users/89edce365b9f41648655523e9d7c68d5/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:17.219939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.242936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:17.244330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:17.275068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:17.282178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5242415-3b3f-4435-bf2c-d3417071b301 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.287911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.288754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.289105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.289446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.332065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 644/1298] 10.222.0.23 () {58 vars in 1058 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:17.332882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.343821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f9c5b-628b-4cd8-930c-0136b5e6b0c0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.351998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.352613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.352920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.353187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.367562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.369175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.379583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 655/1299] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:17.386266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.404988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a28e0b-6523-4dbd-a3a3-1a937e8ed2e7 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.410473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad4e85f-660d-4b50-9d3f-bf29cb7be1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.411464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4e85f-660d-4b50-9d3f-bf29cb7be1e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.411746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4e85f-660d-4b50-9d3f-bf29cb7be1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.412162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4e85f-660d-4b50-9d3f-bf29cb7be1e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.413167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad4e85f-660d-4b50-9d3f-bf29cb7be1e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.415746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad4e85f-660d-4b50-9d3f-bf29cb7be1e9 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.423086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.424681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.435719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 645/1300] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:17.436884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.448545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5f6f18-5125-4edd-920e-9fdd1e027be4 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.453753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.453941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 656/1301] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/projects => generated 387 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:17.455181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.455181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.455410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.455500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.455587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.456182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.456312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.458355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.494954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] MFA Rules not processed for user `89edce365b9f41648655523e9d7c68d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:17.496368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:17.504645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 646/1302] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/projects => generated 373 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:17.506026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.511802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:17.602825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 657/1303] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:17.602825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.626198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4486c1-b440-48f2-861d-138657d4092b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.633612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.634341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.634515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.634785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.699891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3990719-7dd8-4904-9534-2306089b14c2 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.704393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:17.728986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.730607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.756034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 647/1304] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:17.760118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.782280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.783254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.783703 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.784429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.785468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:17.787839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:17.823552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 658/1305] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/users => generated 300 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:17.823552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:17.906206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b44465f-4d08-46d0-b6f4-a1b49f5b4e67 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:17.910430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:17.911105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:17.911933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:17.912323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:17.997897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194b8a85-112a-48e4-b2e5-ee445ecd036b service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.015641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.016617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.016925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.017195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.017836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.020067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.045566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.048961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.064273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 659/1306] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:17 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:18.071700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 648/1307] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:17 2022] POST /identity/v3/users => generated 293 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:18.072511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.081500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.095566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee034ec7-c6b8-41d8-bddd-1e8102e6e94c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.108140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.108857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.109136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.109457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.110151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.111767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.124966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fea69b-2733-4478-94d9-6d5f032605f7 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.134262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.134899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.135178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.135453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.142684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.145525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.152261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 649/1308] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:18 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:18.157991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.172311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 660/1309] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:18 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:18.175728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.186108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f8014-e77a-4308-98e7-3c99ef1757ad admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.202962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cad523-5982-4908-8911-6110857d3441 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.207254 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.207966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.214204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.214562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] PATH_INFO: `/v3/projects/74536e0d3013450aa31a6eba8afbdd8b/users/297eb23c77cb41c58db4a232887fe0c7/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.215933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.219453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.220042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.220483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] PATH_INFO: `/v3/projects/bd306e1df3f24f0cbbd1291f2aefe5a7/users/1e6a466b0af4425ab362ab836d67fb68/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.261494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74536e0d3013450aa31a6eba8afbdd8b, user_id=297eb23c77cb41c58db4a232887fe0c7, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.264624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.265210 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd306e1df3f24f0cbbd1291f2aefe5a7, user_id=1e6a466b0af4425ab362ab836d67fb68, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.270109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.274567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:18.302056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 650/1310] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:18 2022] PUT /identity/v3/projects/74536e0d3013450aa31a6eba8afbdd8b/users/297eb23c77cb41c58db4a232887fe0c7/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:18.302300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 661/1311] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:18 2022] PUT /identity/v3/projects/bd306e1df3f24f0cbbd1291f2aefe5a7/users/1e6a466b0af4425ab362ab836d67fb68/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:18.302413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.302595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.332331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:18.426158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d9c7a1-ed52-4008-954b-a21a699bcfb4 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.431610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.432398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.432791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.433422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.434048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.435766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.440300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c87ab9b-7b17-440e-91f0-7138210b0da8 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.452744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 651/1312] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:18 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:18.453377 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.454018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.454333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.454606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.455258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.457008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.460567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.472106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd7cc64-e30a-430b-98d0-0b874e59d463 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.484506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 662/1313] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:18 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:18.485663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.491383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d8128-987a-4d78-bc9c-ec8778ff66d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.492426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d8128-987a-4d78-bc9c-ec8778ff66d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.493157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d8128-987a-4d78-bc9c-ec8778ff66d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.494024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d8128-987a-4d78-bc9c-ec8778ff66d5 admin admin] PATH_INFO: `/v3/projects/74536e0d3013450aa31a6eba8afbdd8b/users/297eb23c77cb41c58db4a232887fe0c7/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.499631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d8128-987a-4d78-bc9c-ec8778ff66d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74536e0d3013450aa31a6eba8afbdd8b, user_id=297eb23c77cb41c58db4a232887fe0c7, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.506135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d8128-987a-4d78-bc9c-ec8778ff66d5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.511766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08fe32-b93d-4616-b15b-9ff401a6e10c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.518743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12070270-55bb-4775-aee6-7eb8b27a238b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.519427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12070270-55bb-4775-aee6-7eb8b27a238b admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.519709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12070270-55bb-4775-aee6-7eb8b27a238b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.519987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12070270-55bb-4775-aee6-7eb8b27a238b admin admin] PATH_INFO: `/v3/projects/bd306e1df3f24f0cbbd1291f2aefe5a7/users/1e6a466b0af4425ab362ab836d67fb68/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.528834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12070270-55bb-4775-aee6-7eb8b27a238b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd306e1df3f24f0cbbd1291f2aefe5a7, user_id=1e6a466b0af4425ab362ab836d67fb68, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.531998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12070270-55bb-4775-aee6-7eb8b27a238b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.542614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 652/1314] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:18 2022] PUT /identity/v3/projects/74536e0d3013450aa31a6eba8afbdd8b/users/297eb23c77cb41c58db4a232887fe0c7/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:18.545012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.545363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.545791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.553903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 663/1315] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:18 2022] PUT /identity/v3/projects/bd306e1df3f24f0cbbd1291f2aefe5a7/users/1e6a466b0af4425ab362ab836d67fb68/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:18.563990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.564275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.564538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.603204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] MFA Rules not processed for user `297eb23c77cb41c58db4a232887fe0c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:18.604683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:18.622874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] MFA Rules not processed for user `1e6a466b0af4425ab362ab836d67fb68`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:18.624141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:18.727823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 653/1316] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:18 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:18.740929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.829303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 664/1317] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:18 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:18.864770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:18.866773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7073e7d-3f30-40be-9397-8d6e5753fe20 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.872910 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946cece9-60ec-4291-814a-7e9769a0a872 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.879532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946cece9-60ec-4291-814a-7e9769a0a872 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.880278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946cece9-60ec-4291-814a-7e9769a0a872 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.880820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946cece9-60ec-4291-814a-7e9769a0a872 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.885458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8887691-86e4-4e51-870c-308f12dc647f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:18.896764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c480b8d-7f6b-4b87-a573-278ec3d7810a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:18.899138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c480b8d-7f6b-4b87-a573-278ec3d7810a service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.899386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c480b8d-7f6b-4b87-a573-278ec3d7810a service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.899669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c480b8d-7f6b-4b87-a573-278ec3d7810a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.906445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946cece9-60ec-4291-814a-7e9769a0a872 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.911714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946cece9-60ec-4291-814a-7e9769a0a872 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.915423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c480b8d-7f6b-4b87-a573-278ec3d7810a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:18.918848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c480b8d-7f6b-4b87-a573-278ec3d7810a service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:18.941655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 665/1318] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:18 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:18.965094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 654/1319] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:18 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:18.988295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243142a0-6c39-4b64-83a8-c3747ade92c7 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:18.988572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243142a0-6c39-4b64-83a8-c3747ade92c7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:18.988902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243142a0-6c39-4b64-83a8-c3747ade92c7 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:18.991517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 666/1320] 10.222.0.23 () {54 vars in 961 bytes} [Wed Oct 12 02:29:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:19.000132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.001010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.001663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.017684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:19.044802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:19.047215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:19.159429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243142a0-6c39-4b64-83a8-c3747ade92c7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:19.244642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 655/1321] 10.222.0.23 () {58 vars in 1058 bytes} [Wed Oct 12 02:29:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:19.247772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:19.274407 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e30078-d5ed-47e7-b73b-c8a7e7fdc34f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:19.284990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9c1095-d7b3-4c63-a2f4-895ab2073239 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:19.287076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c1095-d7b3-4c63-a2f4-895ab2073239 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.288017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c1095-d7b3-4c63-a2f4-895ab2073239 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.288964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c1095-d7b3-4c63-a2f4-895ab2073239 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.356426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243142a0-6c39-4b64-83a8-c3747ade92c7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:19.367438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:19.369566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.369951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.370440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] PATH_INFO: `/v3/users/077facfd33854415ad33ee217c4f7121` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.373549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] RBAC: Authorizing `identity:delete_user(user_id=077facfd33854415ad33ee217c4f7121)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:19.377969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:19.393730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:19.452219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c1095-d7b3-4c63-a2f4-895ab2073239 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:19.456673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c1095-d7b3-4c63-a2f4-895ab2073239 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:19.480732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 656/1322] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:19.481428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '077facfd33854415ad33ee217c4f7121'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.484077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228580f8-735d-409b-9d95-59d09b6f6e53 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.485459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228580f8-735d-409b-9d95-59d09b6f6e53 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.485651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228580f8-735d-409b-9d95-59d09b6f6e53 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.489699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 657/1323] 10.222.0.23 () {54 vars in 912 bytes} [Wed Oct 12 02:29:19 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:19.491599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228580f8-735d-409b-9d95-59d09b6f6e53 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:19.511278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '077facfd33854415ad33ee217c4f7121'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.513928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '077facfd33854415ad33ee217c4f7121'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.521166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228580f8-735d-409b-9d95-59d09b6f6e53 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:19.527353 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:19.529319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '077facfd33854415ad33ee217c4f7121'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.530018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5779bc1f-d0f7-488a-8cb7-753191d61431 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '077facfd33854415ad33ee217c4f7121'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.531926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.532220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.533608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] PATH_INFO: `/v3/users/93727a50124b4ac995770d38ef23e43f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.539828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] RBAC: Authorizing `identity:delete_user(user_id=93727a50124b4ac995770d38ef23e43f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:19.541580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: WARNING py.warnings [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:29:19.541580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: warnings.warn(msg) Oct 12 02:29:19.541580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: Oct 12 02:29:19.544419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:19.557569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 667/1324] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:19 2022] DELETE /identity/v3/users/077facfd33854415ad33ee217c4f7121 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:19.559443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dae3ce2-9694-40ff-a000-029c3c6b29ed None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.562541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dae3ce2-9694-40ff-a000-029c3c6b29ed None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.562777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dae3ce2-9694-40ff-a000-029c3c6b29ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.610621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0dae3ce2-9694-40ff-a000-029c3c6b29ed None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:19.611773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0dae3ce2-9694-40ff-a000-029c3c6b29ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:19.695957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93727a50124b4ac995770d38ef23e43f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.723129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93727a50124b4ac995770d38ef23e43f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.724033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93727a50124b4ac995770d38ef23e43f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.736873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93727a50124b4ac995770d38ef23e43f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.737612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53a36760-2a88-4208-80bf-84a1a90cd97b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93727a50124b4ac995770d38ef23e43f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:19.767228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 658/1325] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:19 2022] DELETE /identity/v3/users/93727a50124b4ac995770d38ef23e43f => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:19.769074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.769466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.769778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.778799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 668/1326] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:29:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:19.783415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:19.785016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:19.785666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:19.803158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:19.816275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:19.817587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:19.839939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:19.845021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:19.951798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 659/1327] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:19.951798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:19.952618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:20.004092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 669/1328] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:20.005039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:20.007887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:20.041463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01059d35-1687-44d1-9c11-cf8ce7aaee48 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:20.047379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:20.048075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.048363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.048665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.049835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:20.140342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489c8e8c-632e-4d29-a20d-6dd3d75e2f1b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:20.159042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5257abeb-2a16-47b4-b06d-e7e105261709 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:20.159967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5257abeb-2a16-47b4-b06d-e7e105261709 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.160408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5257abeb-2a16-47b4-b06d-e7e105261709 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.161018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5257abeb-2a16-47b4-b06d-e7e105261709 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.166222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:20.168201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:20.181868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5257abeb-2a16-47b4-b06d-e7e105261709 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:20.183814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5257abeb-2a16-47b4-b06d-e7e105261709 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:20.186747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 660/1329] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:20.202448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:20.204797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 670/1330] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:20.228585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d5f182-980f-4052-85c2-cb754101028c service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:20.244439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942ee28e-2fe6-434d-8967-a0c03b78b227 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:20.246804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ee28e-2fe6-434d-8967-a0c03b78b227 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.247089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ee28e-2fe6-434d-8967-a0c03b78b227 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.247364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ee28e-2fe6-434d-8967-a0c03b78b227 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.273746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942ee28e-2fe6-434d-8967-a0c03b78b227 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:20.276079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942ee28e-2fe6-434d-8967-a0c03b78b227 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:20.300527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 661/1331] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:20.452579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04767623-e4ec-4f32-b889-5db54a72099a None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.453067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04767623-e4ec-4f32-b889-5db54a72099a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.453434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04767623-e4ec-4f32-b889-5db54a72099a None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.455100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 671/1332] 10.222.0.23 () {54 vars in 912 bytes} [Wed Oct 12 02:29:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:20.461246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.461559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.461798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.494684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:20.497185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:20.513499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 662/1333] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:29:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:20.522709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04767623-e4ec-4f32-b889-5db54a72099a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:20.524494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-04767623-e4ec-4f32-b889-5db54a72099a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:20.540922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04767623-e4ec-4f32-b889-5db54a72099a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:20.546035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba4eec5-15d3-4f32-b83e-5cff914901de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:20.546674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba4eec5-15d3-4f32-b83e-5cff914901de service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.546987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba4eec5-15d3-4f32-b83e-5cff914901de service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.547262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba4eec5-15d3-4f32-b83e-5cff914901de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.564734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba4eec5-15d3-4f32-b83e-5cff914901de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:20.566723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba4eec5-15d3-4f32-b83e-5cff914901de service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:20.593764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 672/1334] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:20.743232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:20.746531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:20.789607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf301c15-1f78-43c2-bbc6-7dc2751ce83a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:20.796040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:20.797279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.797789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.798259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] PATH_INFO: `/v3/users/f428a9f1b0834b89b4a87e48469d224a` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:20.799667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f428a9f1b0834b89b4a87e48469d224a)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:20.803250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:20.854829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:20.895423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f428a9f1b0834b89b4a87e48469d224a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:20.918703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f428a9f1b0834b89b4a87e48469d224a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:20.919938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f428a9f1b0834b89b4a87e48469d224a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:20.930699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f428a9f1b0834b89b4a87e48469d224a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:20.931938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f428a9f1b0834b89b4a87e48469d224a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:20.979912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 663/1335] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:20 2022] DELETE /identity/v3/users/f428a9f1b0834b89b4a87e48469d224a => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:20.982698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:20.983624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:20.984023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:21.043769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:21.045055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:21.202564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 673/1336] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:21.239321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:21.241472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:21.281412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:21.282987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:21.307733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f76ab-e404-43a1-a9ee-d0225427b0c9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:21.325024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:21.326126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:21.327112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:21.327598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] PATH_INFO: `/v3/projects/ccda4a4b3d95486caaeb607896fdaced` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:21.329037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccda4a4b3d95486caaeb607896fdaced)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:21.330323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: WARNING py.warnings [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 12 02:29:21.330323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: warnings.warn(msg) Oct 12 02:29:21.330323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: Oct 12 02:29:21.332035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:21.462070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dad1ea-9fc3-4949-99fa-87686f5001c9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:21.462070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:21.462070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:21.462070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:21.486884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:21.486884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:21.486884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:21.496304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 664/1337] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:21.538161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccda4a4b3d95486caaeb607896fdaced', 'request_id': 'req-564e42e1-3461-4381-b681-a5f6b5d9e939'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.565452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccda4a4b3d95486caaeb607896fdaced', 'request_id': 'req-564e42e1-3461-4381-b681-a5f6b5d9e939'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.619453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccda4a4b3d95486caaeb607896fdaced', 'request_id': 'req-564e42e1-3461-4381-b681-a5f6b5d9e939'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.657050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] The token cache is being invalidate because project ccda4a4b3d95486caaeb607896fdaced was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:21.657368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.658448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.661600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 674/1338] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:21 2022] DELETE /identity/v3/projects/ccda4a4b3d95486caaeb607896fdaced => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:21.666006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:21.678975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14734d2d-b834-4cd2-8255-b0ab8f03818d service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:21.799547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:21.800299 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:21.800607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:21.800848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] PATH_INFO: `/v3/users/89edce365b9f41648655523e9d7c68d5` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:21.803503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=89edce365b9f41648655523e9d7c68d5)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:21.806623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:21.925099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:21.934324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89edce365b9f41648655523e9d7c68d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.965802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89edce365b9f41648655523e9d7c68d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.966580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89edce365b9f41648655523e9d7c68d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.984091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89edce365b9f41648655523e9d7c68d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:21.984091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89edce365b9f41648655523e9d7c68d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.007932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 665/1339] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:21 2022] DELETE /identity/v3/users/89edce365b9f41648655523e9d7c68d5 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:22.128029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:22.129533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:22.150778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564e42e1-3461-4381-b681-a5f6b5d9e939 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:22.218927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:22.270487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:22.272173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:22.272173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:22.272396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] PATH_INFO: `/v3/projects/fe0a12a6b1564071955fe9c37fd5534c` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:22.286035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe0a12a6b1564071955fe9c37fd5534c)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:22.288332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:22.387156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe0a12a6b1564071955fe9c37fd5534c', 'request_id': 'req-888cb6ac-4dd9-428e-84ee-bfa456c7d900'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.394100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe0a12a6b1564071955fe9c37fd5534c', 'request_id': 'req-888cb6ac-4dd9-428e-84ee-bfa456c7d900'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.407069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe0a12a6b1564071955fe9c37fd5534c', 'request_id': 'req-888cb6ac-4dd9-428e-84ee-bfa456c7d900'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.432914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] The token cache is being invalidate because project fe0a12a6b1564071955fe9c37fd5534c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:22.433194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.441222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.446434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 675/1340] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:22 2022] DELETE /identity/v3/projects/fe0a12a6b1564071955fe9c37fd5534c => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:22.447480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:22.448978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:22.474372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac6bf52-3062-4931-b55a-c0b0e0977d2f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:22.591687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:22.592409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:22.592649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:22.593931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] PATH_INFO: `/v3/users/1e6a466b0af4425ab362ab836d67fb68` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:22.594094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e6a466b0af4425ab362ab836d67fb68)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:22.596059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:22.696946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e6a466b0af4425ab362ab836d67fb68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.749143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e6a466b0af4425ab362ab836d67fb68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.752521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e6a466b0af4425ab362ab836d67fb68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.768319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e6a466b0af4425ab362ab836d67fb68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.770147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:22.770312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9004b5b9-49d6-46e9-839d-733a357f795a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e6a466b0af4425ab362ab836d67fb68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.772505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:22.798627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888cb6ac-4dd9-428e-84ee-bfa456c7d900 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:22.814680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:22.816147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:22.816424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:22.816767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] PATH_INFO: `/v3/projects/2d2ef7ec625d4fd79045b3d5001dc790` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:22.818016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d2ef7ec625d4fd79045b3d5001dc790)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:22.820234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:22.823563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 666/1341] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:22 2022] DELETE /identity/v3/users/1e6a466b0af4425ab362ab836d67fb68 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:22.914608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d2ef7ec625d4fd79045b3d5001dc790', 'request_id': 'req-7b902e18-8f3a-482f-8734-0e6332a76500'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.921498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d2ef7ec625d4fd79045b3d5001dc790', 'request_id': 'req-7b902e18-8f3a-482f-8734-0e6332a76500'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.939444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d2ef7ec625d4fd79045b3d5001dc790', 'request_id': 'req-7b902e18-8f3a-482f-8734-0e6332a76500'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.975605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] The token cache is being invalidate because project 2d2ef7ec625d4fd79045b3d5001dc790 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:22.975912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.977955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7b902e18-8f3a-482f-8734-0e6332a76500 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:22.980890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 676/1342] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:22 2022] DELETE /identity/v3/projects/2d2ef7ec625d4fd79045b3d5001dc790 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:22.985507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:22.985793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:22.986070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.010418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:23.011661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:23.081119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:23.123055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 667/1343] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:23.141222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.142186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.142739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.178825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:23.180288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:23.192575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 677/1344] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:23.204781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.235371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fabf992-352f-4819-8205-8a29dc991ee2 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.241984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.242766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.243134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.243515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.244419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:23.244878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.246960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.275101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 668/1345] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:23.279338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.292333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b9ff36-0e0b-401e-a5f6-8edb71239ad0 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.306887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.307580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.308383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.308910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.309798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.311668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.314036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:23.331283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.346829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e7a91-90a5-4fb1-a3ba-6521ec265bcd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.363933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.364763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.365467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.366121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] PATH_INFO: `/v3/projects/a0fc9e9f616e45eab32cc99f24af1bee` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.368169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0fc9e9f616e45eab32cc99f24af1bee)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.372289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.403924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 678/1346] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:23 2022] POST /identity/v3/projects => generated 403 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:23.406559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.460064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:23.535768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0fc9e9f616e45eab32cc99f24af1bee', 'request_id': 'req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:23.545029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0fc9e9f616e45eab32cc99f24af1bee', 'request_id': 'req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:23.554844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0fc9e9f616e45eab32cc99f24af1bee', 'request_id': 'req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:23.570760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] The token cache is being invalidate because project a0fc9e9f616e45eab32cc99f24af1bee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:23.571508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:23.572364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7a98ce9a-60fc-4fc1-ab67-c089be6cf5f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:23.574606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 669/1347] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:23 2022] DELETE /identity/v3/projects/a0fc9e9f616e45eab32cc99f24af1bee => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:23.579166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f539d-108c-47a3-8f82-32b1c57689b0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.579698 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f539d-108c-47a3-8f82-32b1c57689b0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.580151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f539d-108c-47a3-8f82-32b1c57689b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.599581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba589ab8-a72f-4429-98f6-29684687d03b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.617683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ee0f539d-108c-47a3-8f82-32b1c57689b0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:23.619220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ee0f539d-108c-47a3-8f82-32b1c57689b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:23.756339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.756517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.756738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.757018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.757669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.759645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.761268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 670/1348] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:23.763062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.763343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.763614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.798226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:23.799584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:23.809208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 679/1349] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:23 2022] POST /identity/v3/users => generated 302 bytes in 403 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:23.813192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.815138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:23.818078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 671/1350] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:23.822771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.835788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1c5e58-1bd4-4ac6-98d2-d03eb77e2059 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.845598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.846007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.846284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.846552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.847284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:23.847643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.849113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733e8d55-6ebf-41b3-b98a-b4614b6ad2da admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.850614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.863589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.864531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.865003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.865728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.866670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.868484 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.874358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 672/1351] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:23.875512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.902677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d7b94-6802-496b-aab1-de08d8d0740a admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.908886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 680/1352] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:23 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:23.916652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:23.919510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.920451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.921069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.921668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.922717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.926334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:23.949565 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41385d59-cea6-463e-a4e2-6c47b747ede7 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:23.970419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:23.971068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:23.971350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:23.971623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:23.972276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:23.973986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.025672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 681/1353] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:23 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:24.028318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.034623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 673/1354] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:23 2022] POST /identity/v3/projects => generated 369 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:24.036349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.067887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136277a5-6cfe-4b9a-b0c2-17cd6aa11b23 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.073574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:24.202618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df33a25-690b-4e72-81f0-72f00d3b661f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.212098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:24.213158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.213643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.214744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.215928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:24.217531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94388b1-51a5-4f59-a75d-ab1286035ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:24.218236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94388b1-51a5-4f59-a75d-ab1286035ce4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.218506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94388b1-51a5-4f59-a75d-ab1286035ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.218776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94388b1-51a5-4f59-a75d-ab1286035ce4 admin admin] PATH_INFO: `/v3/projects/497fa1bde80f493785e67e9ce21a35ab/users/e6d10e2c7de744c289d0bf46e7f0fa9f/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.223934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.254766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94388b1-51a5-4f59-a75d-ab1286035ce4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=497fa1bde80f493785e67e9ce21a35ab, user_id=e6d10e2c7de744c289d0bf46e7f0fa9f, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:24.258089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94388b1-51a5-4f59-a75d-ab1286035ce4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.287220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 682/1355] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:24 2022] PUT /identity/v3/projects/497fa1bde80f493785e67e9ce21a35ab/users/e6d10e2c7de744c289d0bf46e7f0fa9f/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:24.299090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.299405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.299695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.301165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 674/1356] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:24 2022] POST /identity/v3/users => generated 285 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:24.306277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.332221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33840713-3b81-4080-9e86-ee3215f52099 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.338883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:24.342022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:24.370950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:24.464888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:24.465638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.465930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.466205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.466860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:24.468934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.482733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 675/1357] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:24 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:24.484323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 683/1358] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:24.486251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.499870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469691d-5b60-494e-97b0-463e6d0496fb admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.506458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:24.507346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.507907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.508592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.509288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:24.512294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.529248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 676/1359] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:24 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:24.531039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.547784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.549618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b697a17-79f3-421a-bb17-3502f990e265 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.551470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:24.562888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d525494f-975e-4bc6-82bd-3a906f1a96cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:24.563576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d525494f-975e-4bc6-82bd-3a906f1a96cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.567503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d525494f-975e-4bc6-82bd-3a906f1a96cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.567927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d525494f-975e-4bc6-82bd-3a906f1a96cd admin admin] PATH_INFO: `/v3/projects/cd6bef0888ea400f91bda74c5004cfa4/users/2c2e2c734b474e9a93ddc818ff60afbe/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.608434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d525494f-975e-4bc6-82bd-3a906f1a96cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd6bef0888ea400f91bda74c5004cfa4, user_id=2c2e2c734b474e9a93ddc818ff60afbe, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:24.614367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d525494f-975e-4bc6-82bd-3a906f1a96cd admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.652077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 684/1360] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:24 2022] PUT /identity/v3/projects/cd6bef0888ea400f91bda74c5004cfa4/users/2c2e2c734b474e9a93ddc818ff60afbe/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:24.662769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.663269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.663552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.729664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1c95c-e876-4357-b159-34a7a8cdfdaf admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.753502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] MFA Rules not processed for user `2c2e2c734b474e9a93ddc818ff60afbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:24.754622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:24.818854 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:24.838013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:24.838847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:24.839312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:24.839591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:24.956277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 685/1361] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:24 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:24.957165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:24.975115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:24.976605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:24.991962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05fa7dc-b9a9-4628-b426-10cd73c14635 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:24.993254 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 677/1362] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:25.003236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:25.014727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.014866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.042222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.053377 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:25.057262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:25.162629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 686/1363] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:24 2022] POST /identity/v3/projects => generated 369 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:25.165852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:25.206209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668e1274-12d9-4250-ba7e-b60741805ff4 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:25.402302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:25.408666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.410105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.410959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.417513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:25.433701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:25.461002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:25.465754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:25.495672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee470dab-d371-4279-8fbc-cb3f405c1008 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:25.525156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:25.525724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.525998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.526264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] PATH_INFO: `/v3/projects/bd306e1df3f24f0cbbd1291f2aefe5a7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.535122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd306e1df3f24f0cbbd1291f2aefe5a7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:25.536172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 678/1364] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:25 2022] POST /identity/v3/users => generated 291 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:29:25.546040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:25.548124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:25.585915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586b6d29-d925-4d5e-8e7c-bacd88046f27 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:25.599238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:25.600571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.601491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.601491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.601932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:25.603417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:25.627268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 679/1365] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:25 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:25.632812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:25.657829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b6a587-7684-495d-bd9e-8395a0af4e65 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:25.670049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd306e1df3f24f0cbbd1291f2aefe5a7', 'request_id': 'req-6eca8ec6-140b-47b0-9594-1b77c17bad60'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:25.674425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:25.686804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd306e1df3f24f0cbbd1291f2aefe5a7', 'request_id': 'req-6eca8ec6-140b-47b0-9594-1b77c17bad60'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:25.702964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd306e1df3f24f0cbbd1291f2aefe5a7', 'request_id': 'req-6eca8ec6-140b-47b0-9594-1b77c17bad60'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:25.727792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] The token cache is being invalidate because project bd306e1df3f24f0cbbd1291f2aefe5a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:25.727792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:25.729061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6eca8ec6-140b-47b0-9594-1b77c17bad60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:25.736872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 687/1366] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:25 2022] DELETE /identity/v3/projects/bd306e1df3f24f0cbbd1291f2aefe5a7 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:25.738621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb6147-a2a1-4c81-98d9-e364a6eb14d8 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.738878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb6147-a2a1-4c81-98d9-e364a6eb14d8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.740756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb6147-a2a1-4c81-98d9-e364a6eb14d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.800561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a7eb6147-a2a1-4c81-98d9-e364a6eb14d8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:25.806334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a7eb6147-a2a1-4c81-98d9-e364a6eb14d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:25.823334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:25.825228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.826807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.827158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] PATH_INFO: `/v3/projects/38b14ab515324f7db68cc214ee230e7c/users/36612aa621e34cd6b7bd84bc7b1debac/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.833104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 688/1367] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:25.842802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:25.843207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:25.843549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:25.873765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=38b14ab515324f7db68cc214ee230e7c, user_id=36612aa621e34cd6b7bd84bc7b1debac, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:25.888646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:25.898645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:25.902090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:25.904068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:25.918106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 689/1368] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:25.922048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:25.923832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 680/1369] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:25 2022] PUT /identity/v3/projects/38b14ab515324f7db68cc214ee230e7c/users/36612aa621e34cd6b7bd84bc7b1debac/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:25.928759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:25.932915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:25.939908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69425d5-9e26-4e48-ae4a-2fe9b403ca09 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.061211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.061853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.062133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.062409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.063075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:26.063380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.065051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.075020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87890887-4a34-44ab-8463-336ecc862c7e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.090004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.090555 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.090838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.091109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.091760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.101244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.108822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 690/1370] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:29:26.110838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.133279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 681/1371] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:25 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:26.135917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059ffcd0-c744-485e-a6c1-fb79e3856a30 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.137434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.150017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e130f4a-1515-4846-b321-0e344f8a3905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.150942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e130f4a-1515-4846-b321-0e344f8a3905 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.151225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e130f4a-1515-4846-b321-0e344f8a3905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.151940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e130f4a-1515-4846-b321-0e344f8a3905 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.153399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e130f4a-1515-4846-b321-0e344f8a3905 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.155958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e130f4a-1515-4846-b321-0e344f8a3905 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.160075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bae6fc-a73f-43ac-954b-09fe6b10dcd6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.173513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.176448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.176758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.179674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] PATH_INFO: `/v3/projects/38b14ab515324f7db68cc214ee230e7c/users/36612aa621e34cd6b7bd84bc7b1debac/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.185294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=38b14ab515324f7db68cc214ee230e7c, user_id=36612aa621e34cd6b7bd84bc7b1debac, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.198548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.232420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 682/1372] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:26 2022] PUT /identity/v3/projects/38b14ab515324f7db68cc214ee230e7c/users/36612aa621e34cd6b7bd84bc7b1debac/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:26.238809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.239266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 691/1373] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:26 2022] POST /identity/v3/projects => generated 373 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:26.243662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.243954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.246156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.332364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] MFA Rules not processed for user `36612aa621e34cd6b7bd84bc7b1debac`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:26.334777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:26.402906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbb21c-04ea-432e-bf6f-7a235f8e37aa admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.510962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.511872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.512253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.512752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.513164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 692/1374] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:26 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:26.513591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.515814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.534202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.535560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:26.583573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 683/1375] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:26 2022] POST /identity/v3/users => generated 287 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:26.592689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.610755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:26.636429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2fa212-856c-4344-a5ee-2613c2e29251 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.648817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.649644 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.649928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.650205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.650862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.661379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.678526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c874d8d6-cf53-4e1c-9e1f-78666bb9e8c9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.679874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 684/1376] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:26 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:26.686103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.686301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.686739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.687033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.687283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.694491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:26.711596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.713171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.715612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd00f4dd-0d1b-414f-b302-66b3aae9d599 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.725831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.732527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 693/1377] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:26 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:26.753642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.754065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.756934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.758683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.762247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.806765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 685/1378] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:26 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:26.807005 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.834369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f123d3-de03-4d83-9dd1-c904cf7b70ab service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.848167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:26.850789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:26.852387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af88896f-6018-4a7a-a039-b5e9127ad313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.853108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af88896f-6018-4a7a-a039-b5e9127ad313 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.853383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af88896f-6018-4a7a-a039-b5e9127ad313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.853650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af88896f-6018-4a7a-a039-b5e9127ad313 admin admin] PATH_INFO: `/v3/projects/f8357cee793144d5bc686d856ea06bf3/users/0833ef4487134fe2bbb617b3f9cc128a/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.876445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a877cef-a299-4b15-9ebe-58c19eb5d962 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:26.886787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:26.889100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af88896f-6018-4a7a-a039-b5e9127ad313 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8357cee793144d5bc686d856ea06bf3, user_id=0833ef4487134fe2bbb617b3f9cc128a, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:26.889848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.890130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.890406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:26.891792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af88896f-6018-4a7a-a039-b5e9127ad313 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:26.897711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:26.933108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 694/1379] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:26 2022] PUT /identity/v3/projects/f8357cee793144d5bc686d856ea06bf3/users/0833ef4487134fe2bbb617b3f9cc128a/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:26.935726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7de746c-70e7-4135-b35e-224da2576d87 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:26.936007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7de746c-70e7-4135-b35e-224da2576d87 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:26.936274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7de746c-70e7-4135-b35e-224da2576d87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:27.006535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a7de746c-70e7-4135-b35e-224da2576d87 None None] MFA Rules not processed for user `0833ef4487134fe2bbb617b3f9cc128a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:27.006535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a7de746c-70e7-4135-b35e-224da2576d87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:27.050552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:27.052224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:27.080123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 686/1380] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:26 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:27.202517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 695/1381] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:27.211319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:27.222801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87298770-567e-4b98-9f47-9b2983f516f8 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:27.362913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:27.362913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:27.362913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:27.362913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:27.384440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:27.390400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:27.438280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 687/1382] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:30.610360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178c0b4b-9fb4-4033-995f-ed17163ff50e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:30.611783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178c0b4b-9fb4-4033-995f-ed17163ff50e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:30.612392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178c0b4b-9fb4-4033-995f-ed17163ff50e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:30.731020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-178c0b4b-9fb4-4033-995f-ed17163ff50e None None] MFA Rules not processed for user `e6d10e2c7de744c289d0bf46e7f0fa9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:30.736365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-178c0b4b-9fb4-4033-995f-ed17163ff50e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:30.935083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 696/1383] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:30 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:30.939801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:30.941287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:31.073298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.073683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.074054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.134370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb3d5b0-4939-44f0-b6ac-2f212cad4fcb service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.144661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] MFA Rules not processed for user `43f16ca13925411fb76a330b6c95395d`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:31.145913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:31.147228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.148998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.149278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.149550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.151917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.155059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.174178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:31.224502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 688/1384] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:30 2022] POST /identity/v3/projects => generated 401 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:31.226969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:31.286409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:31.343121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 697/1385] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:31 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:31.374625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:31.376588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:31.424664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584534be-5387-4ad2-a274-5fac5fb2e734 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.439088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.440016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.440467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.440960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.441753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.444425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.488607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 689/1386] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:31 2022] POST /identity/v3/users => generated 307 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:31.493921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:31.510570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4482c985-b797-47c2-9306-9fdb6711aac7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.523175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.523819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.524099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.524374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.525113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.529914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.559168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 690/1387] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:31 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:31.560766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:31.569981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832592bb-7ef3-4548-8460-e06f4be42400 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.583955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc13052-99be-4431-867c-e73d744c874e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.604772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.605740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.606191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.606617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] PATH_INFO: `/v3/projects/6d9c038f998349f59030b27a69f07e4b/users/4b165df9a9b44d4aadb3a2664b68dcd1/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.647590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d9c038f998349f59030b27a69f07e4b, user_id=4b165df9a9b44d4aadb3a2664b68dcd1, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.655235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.686292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 691/1388] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:31 2022] PUT /identity/v3/projects/6d9c038f998349f59030b27a69f07e4b/users/4b165df9a9b44d4aadb3a2664b68dcd1/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:31.696676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:31.725411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.726008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.726331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.726517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.827670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded0451-0018-4702-94f2-54dc5e57eb23 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.834362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.835599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.835599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.836231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.836578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:31.837129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.839702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.852636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 692/1389] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:31 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:31.857244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:31.859350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.860990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.871069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cbca74-ce61-4bd5-8048-0e10298aebda admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:31.876288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73bb226-165a-4f12-85be-4a67db4beef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:31.877069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73bb226-165a-4f12-85be-4a67db4beef6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:31.877411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73bb226-165a-4f12-85be-4a67db4beef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:31.877746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73bb226-165a-4f12-85be-4a67db4beef6 admin admin] PATH_INFO: `/v3/projects/6d9c038f998349f59030b27a69f07e4b/users/4b165df9a9b44d4aadb3a2664b68dcd1/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:31.880867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73bb226-165a-4f12-85be-4a67db4beef6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d9c038f998349f59030b27a69f07e4b, user_id=4b165df9a9b44d4aadb3a2664b68dcd1, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:31.884240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73bb226-165a-4f12-85be-4a67db4beef6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:31.891289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 698/1390] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:31 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:31.899263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 693/1391] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:31 2022] PUT /identity/v3/projects/6d9c038f998349f59030b27a69f07e4b/users/4b165df9a9b44d4aadb3a2664b68dcd1/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:32.347449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:32.368008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614054b0-0ba9-44e9-a7ff-03eeadafc3fc service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:32.550795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:32.551381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:32.551662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:32.551937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] PATH_INFO: `/v3/users/2c2e2c734b474e9a93ddc818ff60afbe` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:32.555400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c2e2c734b474e9a93ddc818ff60afbe)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:32.569803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:32.710138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c2e2c734b474e9a93ddc818ff60afbe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:32.741108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c2e2c734b474e9a93ddc818ff60afbe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:32.741928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c2e2c734b474e9a93ddc818ff60afbe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:32.772213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c2e2c734b474e9a93ddc818ff60afbe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:32.774079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c2e2c734b474e9a93ddc818ff60afbe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:32.804454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 699/1392] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:32 2022] DELETE /identity/v3/users/2c2e2c734b474e9a93ddc818ff60afbe => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:32.809535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34359935-aac2-47ec-a9be-74a9529a0736 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:32.810055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34359935-aac2-47ec-a9be-74a9529a0736 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:32.810055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34359935-aac2-47ec-a9be-74a9529a0736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:32.844608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-34359935-aac2-47ec-a9be-74a9529a0736 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:32.845891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-34359935-aac2-47ec-a9be-74a9529a0736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:32.941324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:32.972498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 694/1393] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:32.996029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:32.997580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:33.098638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:33.098927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:33.099760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:33.192478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] MFA Rules not processed for user `47f30e76f6894e5f9c06f71adde020e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:33.196937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:33.220572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf5c4db-c225-4b21-9f38-28f1481d8bed admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:33.225348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c9ad5-5124-44fa-b4de-2a116e7be895 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:33.226515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c9ad5-5124-44fa-b4de-2a116e7be895 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:33.227071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c9ad5-5124-44fa-b4de-2a116e7be895 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:33.227507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c9ad5-5124-44fa-b4de-2a116e7be895 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:33.244655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c9ad5-5124-44fa-b4de-2a116e7be895 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:33.247251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c9ad5-5124-44fa-b4de-2a116e7be895 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:33.263368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 700/1394] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:33.369126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 695/1395] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:33.380014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0938450-3174-4176-a99f-090c63d056ff None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:33.380751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0938450-3174-4176-a99f-090c63d056ff None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:33.381215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0938450-3174-4176-a99f-090c63d056ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:33.420669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c0938450-3174-4176-a99f-090c63d056ff None None] MFA Rules not processed for user `47f30e76f6894e5f9c06f71adde020e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:33.424860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c0938450-3174-4176-a99f-090c63d056ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:33.444084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 701/1396] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:33.457493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:33.459738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:33.641977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f869578-c35b-4d52-97fe-c77a0a56e38d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:33.654282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c0918f-083c-46aa-829a-1967724a67ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:33.655024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c0918f-083c-46aa-829a-1967724a67ec service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:33.655405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c0918f-083c-46aa-829a-1967724a67ec service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:33.655782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c0918f-083c-46aa-829a-1967724a67ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:33.666797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c0918f-083c-46aa-829a-1967724a67ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:33.668994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c0918f-083c-46aa-829a-1967724a67ec service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:33.693393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 696/1397] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:33 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:34.272543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:34.273097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:34.273097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:34.282951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:34.339271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] MFA Rules not processed for user `4b165df9a9b44d4aadb3a2664b68dcd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:34.340767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:34.602317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 702/1398] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:34 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:34.676104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c47496b-f3c4-4603-8d92-421f9e9bfe4b None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:34.676614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c47496b-f3c4-4603-8d92-421f9e9bfe4b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:34.676681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:34.676772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c47496b-f3c4-4603-8d92-421f9e9bfe4b None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:34.678167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:34.678391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 697/1399] 10.222.0.23 () {54 vars in 912 bytes} [Wed Oct 12 02:29:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:34.685488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:34.685954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:34.686398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:34.720732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f89652-d2f6-4802-acf7-b2f3eb187cfd None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:34.737069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:34.737730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:34.738001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:34.738272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] PATH_INFO: `/v3/projects/cd6bef0888ea400f91bda74c5004cfa4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:34.747397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd6bef0888ea400f91bda74c5004cfa4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:34.752638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:34.804330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:34.807998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:34.829101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 698/1400] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:29:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:34.862172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:34.874186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ea7a9-d4df-46b9-9fb8-2b7d379df8f9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:34.902830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd6bef0888ea400f91bda74c5004cfa4', 'request_id': 'req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:34.919386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd6bef0888ea400f91bda74c5004cfa4', 'request_id': 'req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:34.929876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd6bef0888ea400f91bda74c5004cfa4', 'request_id': 'req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:34.948831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] The token cache is being invalidate because project cd6bef0888ea400f91bda74c5004cfa4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:34.949141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:34.949792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:34.952520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 703/1401] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:34 2022] DELETE /identity/v3/projects/cd6bef0888ea400f91bda74c5004cfa4 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:34.963584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:34.965533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:35.002319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:35.005516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:35.005810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:35.006080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:35.012771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:35.025458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:35.033662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:35.035129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732c20ff-ce9f-46f3-857e-cbbf44a109ea service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:35.051257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 699/1402] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:35.101887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e5e4dc-99ed-4723-a194-2ff17ca1f8ae admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:35.109385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:35.110698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:35.114531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:35.114531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] PATH_INFO: `/v3/users/36612aa621e34cd6b7bd84bc7b1debac` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:35.114531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=36612aa621e34cd6b7bd84bc7b1debac)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:35.115468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:35.134030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:35.217695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:35.218045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:35.218340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:35.231674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36612aa621e34cd6b7bd84bc7b1debac'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:35.275299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36612aa621e34cd6b7bd84bc7b1debac'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:35.279380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36612aa621e34cd6b7bd84bc7b1debac'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:35.295988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36612aa621e34cd6b7bd84bc7b1debac'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:35.296900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-77e5b712-07e3-4d7b-9694-bc9d1d114e6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36612aa621e34cd6b7bd84bc7b1debac'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:35.301917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] MFA Rules not processed for user `bcd7f1121dab4e8db4c747c51ed49240`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:35.303816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:35.329146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 704/1403] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:34 2022] DELETE /identity/v3/users/36612aa621e34cd6b7bd84bc7b1debac => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:35.486246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 700/1404] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:35 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:35.497336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:35.497984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:35.498612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:35.599698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] MFA Rules not processed for user `bcd7f1121dab4e8db4c747c51ed49240`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:35.603455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:35.775440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 705/1405] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:35 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:35.784136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:35.786536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:35.988981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c987ca6-a484-4af8-8e10-bdcba0bd6afa None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:36.301860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad47fd78-fcd6-4848-91d4-af63d9ccc1e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:36.303041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad47fd78-fcd6-4848-91d4-af63d9ccc1e0 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:36.303316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad47fd78-fcd6-4848-91d4-af63d9ccc1e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:36.303584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad47fd78-fcd6-4848-91d4-af63d9ccc1e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:36.330881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad47fd78-fcd6-4848-91d4-af63d9ccc1e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:36.332499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad47fd78-fcd6-4848-91d4-af63d9ccc1e0 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:36.351827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:36.353605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:36.354970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 701/1406] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:35 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:36.596045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00661eec-2e9c-4b69-b4e8-65ea0bef2d8f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:36.635375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:36.636012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:36.636291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:36.636563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:36.642270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:36.675908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:36.697212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6806b3-8b96-45eb-a4f0-a3683937a26c service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:36.712885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 706/1407] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:37.698604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:37.699514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:37.700038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:37.726196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:37.734111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] MFA Rules not processed for user `43f16ca13925411fb76a330b6c95395d`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:37.735585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:37.745909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332c869f-2c71-4f35-8928-38dc71bfe93a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:37.746792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332c869f-2c71-4f35-8928-38dc71bfe93a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:37.747524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332c869f-2c71-4f35-8928-38dc71bfe93a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:37.797130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-332c869f-2c71-4f35-8928-38dc71bfe93a None None] MFA Rules not processed for user `1fceec891ef24ba3a39da72c90fb733a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:37.799073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-332c869f-2c71-4f35-8928-38dc71bfe93a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:37.834423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:37.898198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 702/1408] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:37 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:37.905700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:37.908106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.003499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 707/1409] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:37 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:38.051598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d01028-5960-4837-bd2f-03b61510a377 None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.051875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d01028-5960-4837-bd2f-03b61510a377 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.052890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d01028-5960-4837-bd2f-03b61510a377 None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.054486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 708/1410] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:29:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:29:38.069807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00dcc26-06c3-4be0-b79c-92eee9382028 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:38.075124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:38.075791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.076071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.076443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] PATH_INFO: `/v3/projects/38b14ab515324f7db68cc214ee230e7c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.077925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b162d-6bcc-4335-8eaf-2e5683ceec30 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.078084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] RBAC: Authorizing `identity:delete_project(project_id=38b14ab515324f7db68cc214ee230e7c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:38.078187 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b162d-6bcc-4335-8eaf-2e5683ceec30 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.078750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b162d-6bcc-4335-8eaf-2e5683ceec30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.080028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:38.153427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-592b162d-6bcc-4335-8eaf-2e5683ceec30 None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:38.156269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-592b162d-6bcc-4335-8eaf-2e5683ceec30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.199192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38b14ab515324f7db68cc214ee230e7c', 'request_id': 'req-12141ad1-07ac-465f-a90d-d28cebb4283a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:38.219855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38b14ab515324f7db68cc214ee230e7c', 'request_id': 'req-12141ad1-07ac-465f-a90d-d28cebb4283a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:38.265915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38b14ab515324f7db68cc214ee230e7c', 'request_id': 'req-12141ad1-07ac-465f-a90d-d28cebb4283a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:38.313633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] The token cache is being invalidate because project 38b14ab515324f7db68cc214ee230e7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:38.314284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:38.316259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12141ad1-07ac-465f-a90d-d28cebb4283a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:38.324544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 703/1411] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:37 2022] DELETE /identity/v3/projects/38b14ab515324f7db68cc214ee230e7c => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:38.331810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c27cd30-c167-4dd5-b5ab-6c5ea7e2e3a8 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.332263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c27cd30-c167-4dd5-b5ab-6c5ea7e2e3a8 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.332691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c27cd30-c167-4dd5-b5ab-6c5ea7e2e3a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.396014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4c27cd30-c167-4dd5-b5ab-6c5ea7e2e3a8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:38.397766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4c27cd30-c167-4dd5-b5ab-6c5ea7e2e3a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.408956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 709/1412] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 332 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:38.470706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:38.501504 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200ba33-deaa-4b77-87ee-1b257c60bece None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.502068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200ba33-deaa-4b77-87ee-1b257c60bece None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.502634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200ba33-deaa-4b77-87ee-1b257c60bece None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.504647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 710/1413] 10.222.0.23 () {54 vars in 961 bytes} [Wed Oct 12 02:29:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:38.510121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2497ba75-cede-4aa2-9f18-d67201aa95ee None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.510699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2497ba75-cede-4aa2-9f18-d67201aa95ee None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.511232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2497ba75-cede-4aa2-9f18-d67201aa95ee None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.513775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 711/1414] 10.222.0.23 () {54 vars in 912 bytes} [Wed Oct 12 02:29:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:29:38.515473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbca877-9480-4296-b1f8-d9b572042004 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.516021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbca877-9480-4296-b1f8-d9b572042004 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.516503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbca877-9480-4296-b1f8-d9b572042004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.589880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 704/1415] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:38.589880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.589880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.589880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.600863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6cbca877-9480-4296-b1f8-d9b572042004 None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:38.609308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6cbca877-9480-4296-b1f8-d9b572042004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.617681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:38.618819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.636476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 712/1416] 10.222.0.23 () {58 vars in 1058 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:38.638004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.646458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.649515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.695564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:38.700324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.739306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 713/1417] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:38.746089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:38.757300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 705/1418] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:38.761717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:38.788404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11ff02f-ec26-41f0-a9c2-e60ae374cfbb None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:38.789564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db2be16-fab8-4dd6-80ab-9303b9f1f7a7 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:38.801670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:38.802999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.803281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.803609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.815783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:38.821500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.823859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.824139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.824861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:38.825165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:38.835605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:38.843115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:38.853907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:38.865716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 714/1419] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:38.866676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:38.868052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:38.882769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d5df70-9411-48df-8221-c6a92657cbf4 service placement] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:38.889293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:38.889974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.890253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.890527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.902370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 706/1420] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:38.903480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:38.912908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:38.914698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:38.928311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26cd82d-0309-4924-a4de-b41b9bd60042 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:38.932397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 715/1421] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:38.937462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:38.938168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:38.938424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:38.938688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:38.939861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:38.941509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:38.982001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 707/1422] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/projects => generated 405 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:38.985083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:39.025885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de3d40-3ff7-47d8-96d3-3248cfbb0c29 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:39.223699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:39.224638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:39.225048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:39.225427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:39.227807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:39.232368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:39.319005 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:39.329635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:39.336851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 716/1423] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:38 2022] POST /identity/v3/users => generated 303 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:39.346083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7dee27-dcee-4926-8fc2-2150906b8916 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:39.361241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:39.362265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:39.362695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:39.363206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:39.363814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:39.366429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:39.392440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:39.402390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 708/1424] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:39 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:39.418005 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a4f196-c770-4a76-99e6-ea85b24d111b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:39.438848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf251d9a-6be1-403a-8ba4-1058c319ac7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:39.441514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf251d9a-6be1-403a-8ba4-1058c319ac7a admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:39.442055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf251d9a-6be1-403a-8ba4-1058c319ac7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:39.442964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf251d9a-6be1-403a-8ba4-1058c319ac7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:39.447848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf251d9a-6be1-403a-8ba4-1058c319ac7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:39.453444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf251d9a-6be1-403a-8ba4-1058c319ac7a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:39.481450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:39.497124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab254ba-67a4-47c4-813a-09eea7cc9fa0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:39.499893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 717/1425] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:39 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:39.513657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:39.527530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:39.528399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:39.529472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] PATH_INFO: `/v3/projects/1c898c7cf4844d2f9041fef8ee635565/users/740e7cb2dfbb48868a5326c8b170d5c3/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:39.561342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c898c7cf4844d2f9041fef8ee635565, user_id=740e7cb2dfbb48868a5326c8b170d5c3, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:39.566323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:39.624876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 709/1426] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:39 2022] PUT /identity/v3/projects/1c898c7cf4844d2f9041fef8ee635565/users/740e7cb2dfbb48868a5326c8b170d5c3/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:39.628728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:39.629206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:39.629877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:39.690320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] MFA Rules not processed for user `740e7cb2dfbb48868a5326c8b170d5c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:39.698810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:39.842067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:39.845970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:40.055932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 718/1427] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:39 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 425 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:40.056716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:40.167839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2278cf-3b25-4621-b225-caac4a00a06b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:40.191688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:40.193754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:40.197435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:40.197812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:40.201186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:40.289824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee9724f-2126-47ed-a0e9-f9dbc395c023 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:40.306909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:40.307696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:40.307971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:40.308287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:40.308955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:40.315380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:40.323130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:40.385334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 719/1428] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:40 2022] POST /identity/v3/projects => generated 409 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:40.388942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:40.496020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:40.500949 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886d06df-25b2-4aed-9ccc-5339b524c2eb service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:40.549572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 710/1429] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 707 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:40.579918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad63c42-12b6-4051-a75c-753dabec191d None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:40.581332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad63c42-12b6-4051-a75c-753dabec191d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:40.581332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad63c42-12b6-4051-a75c-753dabec191d None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:40.590883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 711/1430] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:29:40 2022] GET /identity => generated 270 bytes in 41 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:40.661970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:40.662654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:40.663126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:40.690509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8b4c6-edcd-480f-9f83-18e21e49dba1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:40.717202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8ca518-5d8f-4c6a-94b9-e7443885c534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:40.719716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8ca518-5d8f-4c6a-94b9-e7443885c534 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:40.720081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8ca518-5d8f-4c6a-94b9-e7443885c534 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:40.720427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8ca518-5d8f-4c6a-94b9-e7443885c534 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:40.721361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8ca518-5d8f-4c6a-94b9-e7443885c534 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:40.730374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8ca518-5d8f-4c6a-94b9-e7443885c534 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:40.752577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] MFA Rules not processed for user `38f000e12e7b4cd1b34875f6c0bf9cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:40.760197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:40.769858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:40.803124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 720/1431] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:40 2022] POST /identity/v3/users => generated 311 bytes in 414 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:29:40.808715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:40.809006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:40.809281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:40.908687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:40.910235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:41.090644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 721/1432] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:29:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:41.092082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:41.127906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb3c1c-cb99-4be8-ba50-5d178fbbde88 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:41.141430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:41.142071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:41.142350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:41.142696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:41.143272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:41.148721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:41.177735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 722/1433] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:41 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:41.180989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:41.185482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:41.202921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 712/1434] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:29:40 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 542 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 12 02:29:41.206918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:41.212101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4533e4-a184-4061-abd4-625e1decd7ab admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:41.222624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:41.223288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:41.223568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:41.224380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] PATH_INFO: `/v3/users/e6d10e2c7de744c289d0bf46e7f0fa9f` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:41.229556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6d10e2c7de744c289d0bf46e7f0fa9f)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:41.231527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:41.257445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f9730-337e-4c89-b453-24bfce7fc98d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:41.302197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:41.306635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:41.308088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:41.309554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] PATH_INFO: `/v3/projects/867bfac4b1324378abc9d1b6a85bc07c/users/6527de09c30b4ae5b3d2bb8e15ef7d45/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:41.394562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6d10e2c7de744c289d0bf46e7f0fa9f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:41.426195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6d10e2c7de744c289d0bf46e7f0fa9f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:41.427201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6d10e2c7de744c289d0bf46e7f0fa9f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:41.434804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] RBAC: Authorizing `identity:create_grant(project_id=867bfac4b1324378abc9d1b6a85bc07c, user_id=6527de09c30b4ae5b3d2bb8e15ef7d45, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:41.444168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6d10e2c7de744c289d0bf46e7f0fa9f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:41.444838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:41.445103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6d10e2c7de744c289d0bf46e7f0fa9f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:41.483874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 713/1435] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:41 2022] PUT /identity/v3/projects/867bfac4b1324378abc9d1b6a85bc07c/users/6527de09c30b4ae5b3d2bb8e15ef7d45/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:41.492909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 723/1436] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:41 2022] DELETE /identity/v3/users/e6d10e2c7de744c289d0bf46e7f0fa9f => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:41.495841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:41.500172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:41.505886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:41.537828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:41.608199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:41.746087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0610dfc7-bf46-4500-b275-f9196e40729b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:41.778905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f173018-c5e0-4285-a724-ce6a1ae6f824 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:41.794569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:41.795456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:41.795920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:41.796367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:41.804821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:41.839319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:41.924855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:41.925630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:41.925904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:41.927686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:41.929668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:41.956055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:41.960617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:41.979411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:41.981628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c466ef79-7a92-4bab-bbc3-e3fa61ae4515 service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:41.988173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 714/1437] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 504 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:41.999922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:42.003527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 724/1438] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:29:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:29:42.005425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:42.005716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:42.006053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:42.021463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:42.027023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] MFA Rules not processed for user `bcd7f1121dab4e8db4c747c51ed49240`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:42.028343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:42.224424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 725/1439] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:42 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:42.339158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eac2aa-dac7-423d-8493-e4ffb2216fec service placement] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:42.361508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e641dffa-9d15-4c8e-acaf-53b3442be0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:42.362136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641dffa-9d15-4c8e-acaf-53b3442be0f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:42.362405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641dffa-9d15-4c8e-acaf-53b3442be0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:42.362811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641dffa-9d15-4c8e-acaf-53b3442be0f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:42.363362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e641dffa-9d15-4c8e-acaf-53b3442be0f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:42.368598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e641dffa-9d15-4c8e-acaf-53b3442be0f5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:42.413349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 715/1440] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:41 2022] GET /identity/v3/roles => generated 1420 bytes in 424 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:42.418874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:42.447385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0936b4e8-39d4-45c7-89b2-37aa237ca6e3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:42.471934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:42.477960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:42.482221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:42.482689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] PATH_INFO: `/v3/projects/867bfac4b1324378abc9d1b6a85bc07c/users/6527de09c30b4ae5b3d2bb8e15ef7d45/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:42.493976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=867bfac4b1324378abc9d1b6a85bc07c, user_id=6527de09c30b4ae5b3d2bb8e15ef7d45, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:42.507241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:42.538237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 726/1441] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:42 2022] PUT /identity/v3/projects/867bfac4b1324378abc9d1b6a85bc07c/users/6527de09c30b4ae5b3d2bb8e15ef7d45/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:42.554082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:42.554082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:42.554082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:42.611789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] MFA Rules not processed for user `6527de09c30b4ae5b3d2bb8e15ef7d45`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:42.613657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:42.827026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 716/1442] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:42 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:42.838423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:42.840189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:42.994768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:43.041252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ea13ca-8c10-4f94-8c3d-0177167d8d40 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:43.046669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:43.048951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:43.049094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:43.051245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:43.083604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:43.085105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:43.105088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:43.106625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:43.141426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 727/1443] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:42 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:43.274844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:43.362544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7beef14-3906-484e-9072-aa1094312fc9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:43.385432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:43.386106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:43.395688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:43.397238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] PATH_INFO: `/v3/projects/497fa1bde80f493785e67e9ce21a35ab` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:43.401737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] RBAC: Authorizing `identity:delete_project(project_id=497fa1bde80f493785e67e9ce21a35ab)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:43.408275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:43.544286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '497fa1bde80f493785e67e9ce21a35ab', 'request_id': 'req-825c28ce-25cf-463d-83cf-3b567d354879'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:43.554514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '497fa1bde80f493785e67e9ce21a35ab', 'request_id': 'req-825c28ce-25cf-463d-83cf-3b567d354879'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:43.572731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '497fa1bde80f493785e67e9ce21a35ab', 'request_id': 'req-825c28ce-25cf-463d-83cf-3b567d354879'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:43.590569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:43.609868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a360dcb1-eb32-4e6f-b274-80305d7d7b48 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:43.610091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] The token cache is being invalidate because project 497fa1bde80f493785e67e9ce21a35ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:43.610497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:43.613478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:43.621028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 717/1444] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:43 2022] DELETE /identity/v3/projects/497fa1bde80f493785e67e9ce21a35ab => generated 0 bytes in 536 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:43.625289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:43.627650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:43.744727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:43.745454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:43.745937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:43.746405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:43.753596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:43.830743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c28ce-25cf-463d-83cf-3b567d354879 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:43.847251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:43.848776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:43.850269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:43.850944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] PATH_INFO: `/v3/users/4b165df9a9b44d4aadb3a2664b68dcd1` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:43.856294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b165df9a9b44d4aadb3a2664b68dcd1)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:43.860727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:43.975778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:43.980861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b165df9a9b44d4aadb3a2664b68dcd1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:43.986871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:44.009638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b165df9a9b44d4aadb3a2664b68dcd1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:44.010324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b165df9a9b44d4aadb3a2664b68dcd1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:44.028189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b165df9a9b44d4aadb3a2664b68dcd1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:44.028852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4fe3221d-6ac5-40ab-ae9b-219f2defa558 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b165df9a9b44d4aadb3a2664b68dcd1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:44.040344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:44.060036 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 728/1445] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:43 2022] GET /identity/v3/auth/tokens => generated 3282 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:44.061796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 718/1446] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:43 2022] DELETE /identity/v3/users/4b165df9a9b44d4aadb3a2664b68dcd1 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:44.615953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:44.696209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b5352-51c2-410e-9243-f3cc17d1f04c service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:44.798817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa7825d-0938-4994-8461-273a3b47c53a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:44.799372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa7825d-0938-4994-8461-273a3b47c53a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:44.799950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa7825d-0938-4994-8461-273a3b47c53a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:44.910676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3fa7825d-0938-4994-8461-273a3b47c53a None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:44.921573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3fa7825d-0938-4994-8461-273a3b47c53a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:45.049873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:45.051729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:45.058217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:45.058571 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] PATH_INFO: `/v3/projects/6d9c038f998349f59030b27a69f07e4b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:45.066067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d9c038f998349f59030b27a69f07e4b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:45.083293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:45.359579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 719/1447] 10.222.0.16 () {58 vars in 1015 bytes} [Wed Oct 12 02:29:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 564 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:45.413829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:45.438362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d9c038f998349f59030b27a69f07e4b', 'request_id': 'req-e45b2761-46e2-45d3-8c1e-cb5a5916f945'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:45.462932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d9c038f998349f59030b27a69f07e4b', 'request_id': 'req-e45b2761-46e2-45d3-8c1e-cb5a5916f945'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:45.501215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d9c038f998349f59030b27a69f07e4b', 'request_id': 'req-e45b2761-46e2-45d3-8c1e-cb5a5916f945'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:45.548288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] The token cache is being invalidate because project 6d9c038f998349f59030b27a69f07e4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:45.549894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:45.551061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:45.561087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 729/1448] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:44 2022] DELETE /identity/v3/projects/6d9c038f998349f59030b27a69f07e4b => generated 0 bytes in 948 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:45.570340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ef46a-6999-44f0-ad19-4b8ae043a810 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:45.572231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ef46a-6999-44f0-ad19-4b8ae043a810 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:45.572762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ef46a-6999-44f0-ad19-4b8ae043a810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:45.607129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:45.610663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:45.658637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a83ef46a-6999-44f0-ad19-4b8ae043a810 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:45.662272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a83ef46a-6999-44f0-ad19-4b8ae043a810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:45.851319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45b2761-46e2-45d3-8c1e-cb5a5916f945 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:45.866074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 720/1449] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:45.873458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:45.873889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:45.874314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:45.878099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:45.882757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:45.884234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:45.886037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:45.893430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:45.963467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:45.971864 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:45.972804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:45.974863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:45.989023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 721/1450] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:45.997827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:46.023816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 730/1451] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:46.027388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184fd5fc-7425-4eaa-a2af-d78d19cf66c1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:46.045764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:46.049022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.049385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.050328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.051714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:46.052627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:46.059499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:46.093732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 722/1452] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:46.099490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:46.141332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6506ab39-66de-467b-9b20-7bc3187ecf2d service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:46.167183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:46.167825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.168102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.168372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.174551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:46.185449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:46.292735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 731/1453] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:46 2022] POST /identity/v3/projects => generated 383 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:46.296751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:46.328311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d40df-d4d5-4c60-8751-b86a6aee0772 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:46.508322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:46.509158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.509652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.510110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.511055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:46.514144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:46.579116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 723/1454] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:46 2022] POST /identity/v3/users => generated 292 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:46.583732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:46.599715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57574e-6c44-46c5-8c87-d6a1c9ba768c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:46.613040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:46.613730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.614028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.615310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.617363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:46.620136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:46.643104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 732/1455] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:46 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:46.646832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:46.674967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0913c4-6e2a-4548-b610-ddb7d9e2be9a admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:46.686400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc20124-0534-4228-8854-5ead0fd60b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:46.688237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc20124-0534-4228-8854-5ead0fd60b9e admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.689830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc20124-0534-4228-8854-5ead0fd60b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.691571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc20124-0534-4228-8854-5ead0fd60b9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.693078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc20124-0534-4228-8854-5ead0fd60b9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:46.696982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc20124-0534-4228-8854-5ead0fd60b9e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:46.724479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 724/1456] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:46 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:46.730509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:46.751935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cdd66c-9b1f-4992-8728-da966c9c3351 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:46.760059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:46.761473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.761888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.762172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] PATH_INFO: `/v3/projects/55de18ffdcd84953815671e93732cf9f/users/ecec9f561fd64a2782f218002ac7aee5/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.811279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] RBAC: Authorizing `identity:create_grant(project_id=55de18ffdcd84953815671e93732cf9f, user_id=ecec9f561fd64a2782f218002ac7aee5, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:46.816219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:46.853292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 733/1457] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:46 2022] PUT /identity/v3/projects/55de18ffdcd84953815671e93732cf9f/users/ecec9f561fd64a2782f218002ac7aee5/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:46.865635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:46.865915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:46.866184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:46.885609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:46.940620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] MFA Rules not processed for user `ecec9f561fd64a2782f218002ac7aee5`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:46.942508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:47.204525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 725/1458] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:46 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:47.209403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:47.426562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad5e6fb-055d-4ad2-a957-e8e5e37d733a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:47.432954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:47.434857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:47.435771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:47.436118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:47.437143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:47.439159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:47.491378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:47.508724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 734/1459] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:47 2022] POST /identity/v3/projects => generated 385 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:47.512682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:47.526269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537ef002-2197-46ae-86fd-b774a116f8a5 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:47.630603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:47.630603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:47.630603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:47.630603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:47.631100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:47.633245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:47.689932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 726/1460] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:47 2022] POST /identity/v3/users => generated 299 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:47.693644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:47.710844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f9712a-8389-4b50-b3e0-a440b4b58e6f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:47.727500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d96197-dec1-44d3-afe7-21bf1c0d1286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:47.728283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d96197-dec1-44d3-afe7-21bf1c0d1286 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:47.728566 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d96197-dec1-44d3-afe7-21bf1c0d1286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:47.728968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d96197-dec1-44d3-afe7-21bf1c0d1286 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:47.729696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d96197-dec1-44d3-afe7-21bf1c0d1286 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:47.732182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d96197-dec1-44d3-afe7-21bf1c0d1286 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:47.762833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 735/1461] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:47 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:47.766772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:47.784184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd9d190-4b51-4229-a92f-21ac48c5626d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:47.799367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:47.802599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:47.803772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:47.804577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] PATH_INFO: `/v3/projects/a1b8e40724234498815a2dc703546e05/users/575145fda1c747408e72e225e8567990/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:47.831303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:47.832424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:47.832684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:47.841120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1b8e40724234498815a2dc703546e05, user_id=575145fda1c747408e72e225e8567990, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:47.844479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:47.858681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] MFA Rules not processed for user `1fceec891ef24ba3a39da72c90fb733a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:47.859907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:47.865627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 727/1462] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:47 2022] PUT /identity/v3/projects/a1b8e40724234498815a2dc703546e05/users/575145fda1c747408e72e225e8567990/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:47.868658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:47.948927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 736/1463] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:47.996939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:48.000933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c4390-2e2e-49aa-a52c-e88292f0ec79 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:48.018583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad8fccd-61ed-4c6f-a092-3edb056df961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:48.019188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad8fccd-61ed-4c6f-a092-3edb056df961 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.019468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad8fccd-61ed-4c6f-a092-3edb056df961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.019738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad8fccd-61ed-4c6f-a092-3edb056df961 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.020435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad8fccd-61ed-4c6f-a092-3edb056df961 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:48.022080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad8fccd-61ed-4c6f-a092-3edb056df961 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:48.041596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 728/1464] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:47 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:48.042717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:48.056918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6271c-5f82-4a2c-a2f4-a8bff42ee2d6 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:48.068456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:48.069346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.069500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.069768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] PATH_INFO: `/v3/projects/a1b8e40724234498815a2dc703546e05/users/575145fda1c747408e72e225e8567990/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.071968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1b8e40724234498815a2dc703546e05, user_id=575145fda1c747408e72e225e8567990, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:48.075262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:48.099554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 737/1465] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:48 2022] PUT /identity/v3/projects/a1b8e40724234498815a2dc703546e05/users/575145fda1c747408e72e225e8567990/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:48.118651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef9374e-513d-4b43-8fa4-cfe510cd99df None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.119074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef9374e-513d-4b43-8fa4-cfe510cd99df None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.119479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef9374e-513d-4b43-8fa4-cfe510cd99df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.154753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8ef9374e-513d-4b43-8fa4-cfe510cd99df None None] MFA Rules not processed for user `575145fda1c747408e72e225e8567990`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:48.156191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8ef9374e-513d-4b43-8fa4-cfe510cd99df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:48.262684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 729/1466] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:48 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:48.273735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:48.275541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:48.368765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9c74d1-e663-44f9-8890-59fc85f05c12 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:48.375532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa169271-5f87-4991-81e4-144053d44431 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:48.376588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa169271-5f87-4991-81e4-144053d44431 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.377077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa169271-5f87-4991-81e4-144053d44431 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.377483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa169271-5f87-4991-81e4-144053d44431 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.520072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa169271-5f87-4991-81e4-144053d44431 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:48.522311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa169271-5f87-4991-81e4-144053d44431 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:48.550562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 738/1467] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:48 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:48.711670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041ed200-70ff-4b2e-9bfd-500ea6d6a5f1 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.712119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041ed200-70ff-4b2e-9bfd-500ea6d6a5f1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.712396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041ed200-70ff-4b2e-9bfd-500ea6d6a5f1 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.714408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 730/1468] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:29:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:29:48.737646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.737936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.738208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.770935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:48.772974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:48.816848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:48.868660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 739/1469] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:29:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:48.889161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041ed200-70ff-4b2e-9bfd-500ea6d6a5f1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:48.893038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-041ed200-70ff-4b2e-9bfd-500ea6d6a5f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:48.919660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041ed200-70ff-4b2e-9bfd-500ea6d6a5f1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:48.927424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f79d8b-05e5-4bb1-89a1-bc72e7e94cdc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:48.928035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f79d8b-05e5-4bb1-89a1-bc72e7e94cdc service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:48.928303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f79d8b-05e5-4bb1-89a1-bc72e7e94cdc service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:48.928582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f79d8b-05e5-4bb1-89a1-bc72e7e94cdc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:48.938300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f79d8b-05e5-4bb1-89a1-bc72e7e94cdc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:48.939908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f79d8b-05e5-4bb1-89a1-bc72e7e94cdc service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:48.950568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 731/1470] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:52.681166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:52.683827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:52.837034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e54b3-fda3-4cd7-9cf9-4c7381e263d9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:52.845665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:52.846456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:52.846797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:52.847095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] PATH_INFO: `/v3/users/0833ef4487134fe2bbb617b3f9cc128a` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:52.848721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0833ef4487134fe2bbb617b3f9cc128a)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:52.850854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:52.902848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0833ef4487134fe2bbb617b3f9cc128a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:52.919169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0833ef4487134fe2bbb617b3f9cc128a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:52.920425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0833ef4487134fe2bbb617b3f9cc128a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:52.932280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0833ef4487134fe2bbb617b3f9cc128a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:52.933359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0833ef4487134fe2bbb617b3f9cc128a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:52.967886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 740/1471] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:52 2022] DELETE /identity/v3/users/0833ef4487134fe2bbb617b3f9cc128a => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:52.969588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:52.970161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:52.970161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:52.992124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:52.993626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:53.104762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 732/1472] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:53.117332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:53.119123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:53.194438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:53.207353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43db288a-c6f1-4536-9c80-7ab97e33844c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:53.211109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:53.211697 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:53.211973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:53.212238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:53.221192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:53.222624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:53.232615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 741/1473] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:53.521514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:53.523025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:53.594927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec70dd8-1295-433f-ae69-a4eed7f6e151 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:53.600652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:53.601411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:53.601693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:53.601946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:53.603873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:53.620821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:53.622659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:53.644429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:53.650018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdaf1eb-0253-4abd-a353-29be53330602 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:53.657154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:53.657802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:53.658126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:53.658611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] PATH_INFO: `/v3/projects/f8357cee793144d5bc686d856ea06bf3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:53.667070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8357cee793144d5bc686d856ea06bf3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:53.678871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:53.721769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:53.723913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b84c2cb-4e91-4d38-90ef-003a7c048260 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:53.737681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 733/1474] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:53 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:53.750362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8357cee793144d5bc686d856ea06bf3', 'request_id': 'req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:53.757987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8357cee793144d5bc686d856ea06bf3', 'request_id': 'req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:53.764793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8357cee793144d5bc686d856ea06bf3', 'request_id': 'req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:53.787821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] The token cache is being invalidate because project f8357cee793144d5bc686d856ea06bf3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:53.788246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:53.789023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-74bdf6ee-bb75-4e57-96b6-6bff542b47c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:53.791289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 742/1475] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:53 2022] DELETE /identity/v3/projects/f8357cee793144d5bc686d856ea06bf3 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:53.798025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:53.798428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:53.800870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:53.847005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:53.851174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:53.979370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 734/1476] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:53.988660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:53.990230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:53.990717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.025464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:54.027048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:54.036147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 743/1477] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:54.042085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.058920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acead60-be8b-4a6f-a730-ba3664031450 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.072803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.073603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.073980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.074351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.075358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:54.075812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.077756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.088100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.089591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:54.100719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 735/1478] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:29:54.103073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.112431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153f2838-4ed7-4e08-93a5-fbb30a1613b0 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.118920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bf308e-035d-422d-aa8b-476795ba9ed6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.123810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.124473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.124775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.125040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] PATH_INFO: `/v3/users/740e7cb2dfbb48868a5326c8b170d5c3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.126682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=740e7cb2dfbb48868a5326c8b170d5c3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.126909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.128096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.128979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.130025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.131637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.132108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.134255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.227149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 736/1479] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:54 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:29:54.242436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.293212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '740e7cb2dfbb48868a5326c8b170d5c3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:54.310970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '740e7cb2dfbb48868a5326c8b170d5c3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:54.311771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '740e7cb2dfbb48868a5326c8b170d5c3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:54.324961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '740e7cb2dfbb48868a5326c8b170d5c3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:54.327165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d2a9c92a-3a19-43fe-81d7-874deca4da3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '740e7cb2dfbb48868a5326c8b170d5c3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:54.350887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 744/1480] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:54 2022] DELETE /identity/v3/users/740e7cb2dfbb48868a5326c8b170d5c3 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:54.360730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.360730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.360730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.396556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7d36db-e850-43d2-a17f-e5341e056c5c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.453260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:54.458011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:54.493707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:54.560381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.561537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.561820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.562091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.562933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.565232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.567153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:54.595095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 737/1481] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:54 2022] POST /identity/v3/users => generated 293 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:54.599525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.601981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:54.617265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04eab60-f90d-4f7e-917b-f94781475a43 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.621217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 745/1482] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:54.627153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.628262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.628806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.629173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.630477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.632199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.634839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.636453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:54.644525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 738/1483] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:54 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:54.648662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.659236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35fdf1-dfc9-450f-b965-293607301fe7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.666568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.667264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.667617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.667951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.668658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.670360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.690043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 739/1484] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:54 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:54.695985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.709325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca59288-7aa9-4680-9598-3f215c30f898 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.716396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f834b8-6529-4cd2-8f4d-7a55bd38459e None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:54.718416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd695ac-f8ab-48d5-a47c-0e9c5d63cb7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.719179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd695ac-f8ab-48d5-a47c-0e9c5d63cb7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.719517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd695ac-f8ab-48d5-a47c-0e9c5d63cb7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.719877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd695ac-f8ab-48d5-a47c-0e9c5d63cb7b admin admin] PATH_INFO: `/v3/projects/b52b118db972455c99dd8bc0daabd409/users/e5c6c7eabb5b4c808c5f72d1ade91e9b/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.727690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:54.728055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.729020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.729833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.742136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd695ac-f8ab-48d5-a47c-0e9c5d63cb7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b52b118db972455c99dd8bc0daabd409, user_id=e5c6c7eabb5b4c808c5f72d1ade91e9b, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.744951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd695ac-f8ab-48d5-a47c-0e9c5d63cb7b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.745762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:54.747628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:54.761832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 740/1485] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:54 2022] PUT /identity/v3/projects/b52b118db972455c99dd8bc0daabd409/users/e5c6c7eabb5b4c808c5f72d1ade91e9b/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:54.774352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:54.774599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 746/1486] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:54.775118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:54.775118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:54.806239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] MFA Rules not processed for user `e5c6c7eabb5b4c808c5f72d1ade91e9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:54.807740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:54.885089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 741/1487] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:54 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:54.886202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:54.898575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb83fd7-90b1-4ee7-99f9-7fa221dc6f9b service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.011401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.016692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.017015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.017263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.017994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.029199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.085641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 747/1488] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:54 2022] POST /identity/v3/projects => generated 383 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:29:55.089670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.112332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791d8ad-72dd-4516-8237-4cb3fc11e7bc None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.183364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.185572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:55.228166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.229007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.229279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.229707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.230358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.232296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.244497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:55.297834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 742/1489] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:55 2022] POST /identity/v3/users => generated 292 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:55.300021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.311122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a7ed5e-ff03-4f9e-a0b1-4f03790c8a59 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.315318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787ec6b5-4c92-472a-8a22-80a4001c23ea admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.316267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.317566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.317920 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.318262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.319112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.320971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.322386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.323333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.324185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.324468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] PATH_INFO: `/v3/projects/1c898c7cf4844d2f9041fef8ee635565` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.325850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c898c7cf4844d2f9041fef8ee635565)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.328896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.333244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 743/1490] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:55.336761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.337919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:55.347883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f075c2f9-0eb1-4d2a-9494-c9835dc48871 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.353240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.353918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.354265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.354599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.355341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.357057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.379067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 744/1491] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:55 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:55.385554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.395061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c898c7cf4844d2f9041fef8ee635565', 'request_id': 'req-5ed9c616-afd8-4e55-9f95-0c035406a63d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.397475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6226555d-889c-436b-b6bb-efaabee8e316 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.401541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c898c7cf4844d2f9041fef8ee635565', 'request_id': 'req-5ed9c616-afd8-4e55-9f95-0c035406a63d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.420519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c898c7cf4844d2f9041fef8ee635565', 'request_id': 'req-5ed9c616-afd8-4e55-9f95-0c035406a63d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.437443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] The token cache is being invalidate because project 1c898c7cf4844d2f9041fef8ee635565 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:55.437978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.438724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.443456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 748/1492] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:55 2022] DELETE /identity/v3/projects/1c898c7cf4844d2f9041fef8ee635565 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:55.445604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.447004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:55.510471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc96a9-1180-439c-a40b-0ce96345dd4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.511226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc96a9-1180-439c-a40b-0ce96345dd4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.511575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc96a9-1180-439c-a40b-0ce96345dd4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.511913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc96a9-1180-439c-a40b-0ce96345dd4b admin admin] PATH_INFO: `/v3/projects/169a683e8541425885ca20c9bf605569/users/6715dc20a448424c9240c67ef789bb00/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.528448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc96a9-1180-439c-a40b-0ce96345dd4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=169a683e8541425885ca20c9bf605569, user_id=6715dc20a448424c9240c67ef789bb00, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.532755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed9c616-afd8-4e55-9f95-0c035406a63d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.532889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc96a9-1180-439c-a40b-0ce96345dd4b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.541911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.542665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.543012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.543516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] PATH_INFO: `/v3/users/6527de09c30b4ae5b3d2bb8e15ef7d45` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.545064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6527de09c30b4ae5b3d2bb8e15ef7d45)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.547481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.559637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 745/1493] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:55 2022] PUT /identity/v3/projects/169a683e8541425885ca20c9bf605569/users/6715dc20a448424c9240c67ef789bb00/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:55.571223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.571554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.571830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.606516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] MFA Rules not processed for user `6715dc20a448424c9240c67ef789bb00`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:55.607805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:55.672079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6527de09c30b4ae5b3d2bb8e15ef7d45'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.713598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6527de09c30b4ae5b3d2bb8e15ef7d45'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.714830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6527de09c30b4ae5b3d2bb8e15ef7d45'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.740855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6527de09c30b4ae5b3d2bb8e15ef7d45'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.741511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6527de09c30b4ae5b3d2bb8e15ef7d45'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:55.747175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 746/1494] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:55.750243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.753436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:55.766857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 749/1495] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:29:55 2022] DELETE /identity/v3/users/6527de09c30b4ae5b3d2bb8e15ef7d45 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:55.847014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:55.872872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a83a0c2-f96e-4e75-8ba6-7f677573de91 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:55.945460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:55.946169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:55.946519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:55.946881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:55.947556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:55.949673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:55.980075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 747/1496] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:55 2022] POST /identity/v3/projects => generated 383 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:55.984274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:55.985862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:55.999312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9ac77c-0c5b-425a-98bd-2ac46dcbe661 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.021655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:56.088364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.090403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.090792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.091642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.094165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.099199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.124485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 750/1497] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:55 2022] POST /identity/v3/users => generated 298 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:56.128416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.139222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad85471-9d7a-4a8f-90f4-170d0cbaaa72 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.144041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.144651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.146625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.147028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.147688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.149320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.164476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 748/1498] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:56 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:56.166930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.190583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a3722d-764c-4fe5-ad70-493a79b567cb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.200399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.202834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.214927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.222220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.222506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.222785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] PATH_INFO: `/v3/projects/d6c3e9c7558d424d9555f0a532df3588/users/205e96b023424da3a767d94d1732cc35/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.227768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a94f09-ee92-4025-becd-4875645138e2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.244376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.245087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.245458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.245731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] PATH_INFO: `/v3/projects/867bfac4b1324378abc9d1b6a85bc07c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.249636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=867bfac4b1324378abc9d1b6a85bc07c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.251577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.264576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6c3e9c7558d424d9555f0a532df3588, user_id=205e96b023424da3a767d94d1732cc35, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.270614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.297997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 751/1499] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:56 2022] PUT /identity/v3/projects/d6c3e9c7558d424d9555f0a532df3588/users/205e96b023424da3a767d94d1732cc35/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:56.302360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.334986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '867bfac4b1324378abc9d1b6a85bc07c', 'request_id': 'req-33786098-9004-411a-92db-a97651f8b1f5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:56.343245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '867bfac4b1324378abc9d1b6a85bc07c', 'request_id': 'req-33786098-9004-411a-92db-a97651f8b1f5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:56.355404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '867bfac4b1324378abc9d1b6a85bc07c', 'request_id': 'req-33786098-9004-411a-92db-a97651f8b1f5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:56.375857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] The token cache is being invalidate because project 867bfac4b1324378abc9d1b6a85bc07c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:29:56.376418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:56.377479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-33786098-9004-411a-92db-a97651f8b1f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:29:56.381546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 749/1500] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:29:56 2022] DELETE /identity/v3/projects/867bfac4b1324378abc9d1b6a85bc07c => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:56.386140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013a702c-3755-44ab-8d22-d278fb4baee4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.386293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10000cc6-564a-4349-905b-95f8df344413 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.386701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10000cc6-564a-4349-905b-95f8df344413 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.387108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10000cc6-564a-4349-905b-95f8df344413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.412562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-10000cc6-564a-4349-905b-95f8df344413 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:56.415048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-10000cc6-564a-4349-905b-95f8df344413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.465674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.466295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.466571 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.466839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.467479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.469140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.476157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:56.481249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 752/1501] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:56 2022] GET /identity/v3/roles => generated 1420 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:56.485410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.487110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.494682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 750/1502] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:56.500127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ad0b1a-f048-4153-8f8d-c50253f83e40 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.501985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.502264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.502534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.505064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80236e5-2edd-4e86-94cd-291246a2072d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.505700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80236e5-2edd-4e86-94cd-291246a2072d admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.505972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80236e5-2edd-4e86-94cd-291246a2072d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.506238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80236e5-2edd-4e86-94cd-291246a2072d admin admin] PATH_INFO: `/v3/projects/d6c3e9c7558d424d9555f0a532df3588/users/205e96b023424da3a767d94d1732cc35/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.508114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80236e5-2edd-4e86-94cd-291246a2072d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6c3e9c7558d424d9555f0a532df3588, user_id=205e96b023424da3a767d94d1732cc35, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.511043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80236e5-2edd-4e86-94cd-291246a2072d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.522743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 753/1503] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:56 2022] PUT /identity/v3/projects/d6c3e9c7558d424d9555f0a532df3588/users/205e96b023424da3a767d94d1732cc35/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:56.524229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:56.525628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.533590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.533869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.534137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.558271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:56.569769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] MFA Rules not processed for user `205e96b023424da3a767d94d1732cc35`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:56.571025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.617665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 751/1504] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:56.621838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.631390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5a8988-50c4-48f3-855c-1634973735fe None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.636045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.636650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.636957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.637228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.637877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:29:56.638201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.639763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.649311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 754/1505] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:56 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:29:56.660940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.662626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.665336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 752/1506] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:29:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:29:56.669573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.679861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365b625-5727-49ba-9b52-f4a2733257b6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.685194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.685801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.686079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.686395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.687034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.688965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.719860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 753/1507] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:56 2022] POST /identity/v3/projects => generated 365 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:56.724088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.731246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a806c91-dc19-4b01-86b8-3ec3db154756 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.808833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.809582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.809862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.810159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.812342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:56.833813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318e955-2251-4fca-9797-0e53518476b7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.840077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.840834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.841611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.842145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.842880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.845346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.878950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 754/1508] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:56 2022] POST /identity/v3/users => generated 283 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:56.883631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.892494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.893838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6865e183-7b82-4756-8a8b-e713966e76af admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.894171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.899636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.900302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.900645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.901001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.902463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.903355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.905095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 755/1509] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:29:56.917309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 755/1510] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:56.918502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.927354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:56.932001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871c4012-d481-40cb-9298-722f35511afc service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.937693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2902d51-52df-4958-baa5-46d066e70701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.938386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2902d51-52df-4958-baa5-46d066e70701 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.938665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2902d51-52df-4958-baa5-46d066e70701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.938933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2902d51-52df-4958-baa5-46d066e70701 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.939684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2902d51-52df-4958-baa5-46d066e70701 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:56.941384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2902d51-52df-4958-baa5-46d066e70701 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:56.953185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 756/1511] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:56 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:29:56.958158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:56.972895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7e2244-0eaa-48cd-99f7-fe766facd097 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:56.978866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:56.979533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:56.979941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:56.980356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] PATH_INFO: `/v3/projects/8548a147f75848ffbfb9c0bc7cb2a057/users/e1e2f57533f144f487ee8504895224f8/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:56.997348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] RBAC: Authorizing `identity:create_grant(project_id=8548a147f75848ffbfb9c0bc7cb2a057, user_id=e1e2f57533f144f487ee8504895224f8, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.000248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.016088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 756/1512] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:56 2022] PUT /identity/v3/projects/8548a147f75848ffbfb9c0bc7cb2a057/users/e1e2f57533f144f487ee8504895224f8/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:29:57.026525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.027025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.027217 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.058012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] MFA Rules not processed for user `e1e2f57533f144f487ee8504895224f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:57.059268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:57.130504 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 757/1513] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:57 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:57.135493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.145031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:57.213041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3282274-1806-4f0e-b61e-8a600b310cab admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.218086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.218779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.218987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.219261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.219896 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.221835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.251894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 757/1514] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:29:57 2022] POST /identity/v3/projects => generated 365 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:57.256575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.273323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12218a66-57da-4d6f-971c-37297c69d785 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.330067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.330603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.330878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.331146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.331765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.333768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.356258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 758/1515] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:29:57 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:29:57.360405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.372519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a803b50c-1a42-4c3e-acab-abf4c33fc2c6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.377611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.378306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.378673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.378948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.380161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.381792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.392647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 758/1516] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:57.397023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.407706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f1f84a-fa52-4dc0-b854-07274acdab7b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.412799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.413458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.413755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.414066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] PATH_INFO: `/v3/projects/1f94465a7c7d402b8108eb06f16d6a83/users/ee2133428d24465fb7e40da2c1989b4d/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.434563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f94465a7c7d402b8108eb06f16d6a83, user_id=ee2133428d24465fb7e40da2c1989b4d, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.437535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.452363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 759/1517] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:57 2022] PUT /identity/v3/projects/1f94465a7c7d402b8108eb06f16d6a83/users/ee2133428d24465fb7e40da2c1989b4d/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:57.458643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.469710 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb5f19c-edab-47f0-86fe-e45a74950649 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.527705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afc16b-2960-4ef7-991c-2fd9dd86cd83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.528320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afc16b-2960-4ef7-991c-2fd9dd86cd83 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.528595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afc16b-2960-4ef7-991c-2fd9dd86cd83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.528886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afc16b-2960-4ef7-991c-2fd9dd86cd83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.529595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0afc16b-2960-4ef7-991c-2fd9dd86cd83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.531193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0afc16b-2960-4ef7-991c-2fd9dd86cd83 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.544916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 759/1518] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:29:57 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:29:57.548223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.559317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfeb270-4a04-496e-8757-0d33f435bab6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.564079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.564783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.565063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.565375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] PATH_INFO: `/v3/projects/1f94465a7c7d402b8108eb06f16d6a83/users/ee2133428d24465fb7e40da2c1989b4d/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.567943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f94465a7c7d402b8108eb06f16d6a83, user_id=ee2133428d24465fb7e40da2c1989b4d, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.570726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.581194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 760/1519] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:29:57 2022] PUT /identity/v3/projects/1f94465a7c7d402b8108eb06f16d6a83/users/ee2133428d24465fb7e40da2c1989b4d/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:29:57.591165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.591440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.591707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.621115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] MFA Rules not processed for user `ee2133428d24465fb7e40da2c1989b4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:29:57.622360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:57.697334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 760/1520] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:29:57 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:29:57.710848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.712622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:29:57.753023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:57.773179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd62ca82-18dd-4aad-9d27-4e29b9ed692a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.777539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.778184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.778457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.778722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.787640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.789092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.798483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 761/1521] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:57 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:29:57.881899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:29:57.886672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:29:57.892125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5d56fa-0851-40b0-ae63-ec55163f8618 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:29:57.898014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:29:57.899035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:29:57.899326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:29:57.899597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:29:57.967904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:29:57.969460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:29:57.978420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 761/1522] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:29:57 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:01.274759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:01.309906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0195a-5e57-41f2-a982-639b72e388c9 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:01.324856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:01.325865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:01.326300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:01.326749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:01.345078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:01.495338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:01.497622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:01.511315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 762/1523] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:02.462423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:02.474022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61948e37-a877-42c7-863c-a6dfb9555729 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:02.480778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b84636f-7972-4494-932b-6c4d76d45918 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:02.481577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84636f-7972-4494-932b-6c4d76d45918 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:02.481897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84636f-7972-4494-932b-6c4d76d45918 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:02.482153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84636f-7972-4494-932b-6c4d76d45918 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:02.559760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b84636f-7972-4494-932b-6c4d76d45918 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:02.560895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b84636f-7972-4494-932b-6c4d76d45918 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:02.575851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 762/1524] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:02 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:03.852485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:03.856555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:03.970204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52eec3d9-b82a-4aa2-b501-af880e723993 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:03.977692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:03.980250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:03.980717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:03.981125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] PATH_INFO: `/v3/users/ecec9f561fd64a2782f218002ac7aee5` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:03.982839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecec9f561fd64a2782f218002ac7aee5)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:03.990340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:04.072380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecec9f561fd64a2782f218002ac7aee5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:04.110384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecec9f561fd64a2782f218002ac7aee5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:04.113735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecec9f561fd64a2782f218002ac7aee5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:04.127125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecec9f561fd64a2782f218002ac7aee5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:04.127922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecec9f561fd64a2782f218002ac7aee5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:04.138891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:04.162581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 763/1525] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:03 2022] DELETE /identity/v3/users/ecec9f561fd64a2782f218002ac7aee5 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:04.167114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:04.167745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:04.168919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:04.213373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:04.217465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:04.403550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 763/1526] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:04.445292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:04.446962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:04.577023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f187d91a-acea-4a2c-8405-ad0437410d14 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:04.582419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bded4c0e-5590-4481-a4f6-985a5d221aae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:04.582966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded4c0e-5590-4481-a4f6-985a5d221aae service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:04.583215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded4c0e-5590-4481-a4f6-985a5d221aae service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:04.583526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded4c0e-5590-4481-a4f6-985a5d221aae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:04.595483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bded4c0e-5590-4481-a4f6-985a5d221aae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:04.597345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bded4c0e-5590-4481-a4f6-985a5d221aae service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:04.610250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 764/1527] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:05.107387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:05.108869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.133888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57817735-33bb-4006-86f8-7c6f864349eb None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:05.143167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:05.144343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.145044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.145476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] PATH_INFO: `/v3/users/e1e2f57533f144f487ee8504895224f8` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:05.149159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1e2f57533f144f487ee8504895224f8)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:05.161059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:05.230607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:05.257725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1e2f57533f144f487ee8504895224f8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.278658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1e2f57533f144f487ee8504895224f8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.280973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1e2f57533f144f487ee8504895224f8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.295029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1e2f57533f144f487ee8504895224f8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.295683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2825d7b5-09f4-430d-b13e-59d38bb9a7d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1e2f57533f144f487ee8504895224f8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.318136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 764/1528] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:05 2022] DELETE /identity/v3/users/e1e2f57533f144f487ee8504895224f8 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:05.322394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.323204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.323615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:05.349560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20a6869-0a94-4230-8445-a9be8b85b426 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.350040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20a6869-0a94-4230-8445-a9be8b85b426 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.350481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20a6869-0a94-4230-8445-a9be8b85b426 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:05.354616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:05.356096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.359738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 765/1529] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:30:05 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:30:05.361123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.361795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.361795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:05.416096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:05.418449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.495628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 765/1530] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:05.529425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:05.532220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.608129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 766/1531] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:30:05 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:05.612893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:05.625439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.665259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91136c2-5c5c-4ed5-bd45-1913b113b922 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:05.678737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3289fd59-e99b-47a2-a22c-e1020ec7effe None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:05.682799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:05.686320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.686401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.686720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] PATH_INFO: `/v3/projects/55de18ffdcd84953815671e93732cf9f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:05.691008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55de18ffdcd84953815671e93732cf9f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:05.694327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:05.695229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:05.696007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.696408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.696834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:05.731994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:05.749046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:05.756051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 766/1532] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:05.761096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:05.765278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.811604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fae91e-0f3e-4665-af31-57656023e1f6 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:05.861762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55de18ffdcd84953815671e93732cf9f', 'request_id': 'req-1c2581ec-bc0b-4091-a95f-cfe090ca9757'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.884588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55de18ffdcd84953815671e93732cf9f', 'request_id': 'req-1c2581ec-bc0b-4091-a95f-cfe090ca9757'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.909729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55de18ffdcd84953815671e93732cf9f', 'request_id': 'req-1c2581ec-bc0b-4091-a95f-cfe090ca9757'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.913007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:05.952716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] The token cache is being invalidate because project 55de18ffdcd84953815671e93732cf9f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:05.953286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.955662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:05.961868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 767/1533] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:05 2022] DELETE /identity/v3/projects/55de18ffdcd84953815671e93732cf9f => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:05.971441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:05.976387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:05.996130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:05.996921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:05.997302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:05.997706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:06.000291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:06.030926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:06.037942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:06.057634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:06.085100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 767/1534] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:06.174964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2581ec-bc0b-4091-a95f-cfe090ca9757 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:06.190842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:06.194230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:06.196282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:06.196906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] PATH_INFO: `/v3/users/575145fda1c747408e72e225e8567990` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:06.200340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] RBAC: Authorizing `identity:delete_user(user_id=575145fda1c747408e72e225e8567990)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:06.205443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:06.357237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '575145fda1c747408e72e225e8567990'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:06.387913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '575145fda1c747408e72e225e8567990'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:06.390285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '575145fda1c747408e72e225e8567990'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:06.407105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '575145fda1c747408e72e225e8567990'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:06.409221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '575145fda1c747408e72e225e8567990'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:06.446610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 768/1535] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:05 2022] DELETE /identity/v3/users/575145fda1c747408e72e225e8567990 => generated 0 bytes in 473 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:07.332382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:07.333958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:07.349991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b5f5a3-4747-4bfc-9c58-6341e0aeae2d service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:07.362763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:07.365021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:07.465961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:07.467052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:07.467338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:07.467612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] PATH_INFO: `/v3/users/e5c6c7eabb5b4c808c5f72d1ade91e9b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:07.469386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5c6c7eabb5b4c808c5f72d1ade91e9b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:07.471765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:07.502862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f697e7b-6dc9-4e55-8b3c-5fa5049aa22b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:07.509503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:07.510151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:07.510540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:07.511066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] PATH_INFO: `/v3/projects/8548a147f75848ffbfb9c0bc7cb2a057` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:07.512263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8548a147f75848ffbfb9c0bc7cb2a057)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:07.514518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:07.542839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5c6c7eabb5b4c808c5f72d1ade91e9b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.568362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5c6c7eabb5b4c808c5f72d1ade91e9b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.569124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5c6c7eabb5b4c808c5f72d1ade91e9b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.581994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5c6c7eabb5b4c808c5f72d1ade91e9b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.583083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5c6c7eabb5b4c808c5f72d1ade91e9b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.586870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:07.594271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8548a147f75848ffbfb9c0bc7cb2a057', 'request_id': 'req-29c5f384-6194-4765-a5ed-d639d30cfe57'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.606784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8548a147f75848ffbfb9c0bc7cb2a057', 'request_id': 'req-29c5f384-6194-4765-a5ed-d639d30cfe57'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.636157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 768/1536] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:07 2022] DELETE /identity/v3/users/e5c6c7eabb5b4c808c5f72d1ade91e9b => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:07.636157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:07.637379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:07.649880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8548a147f75848ffbfb9c0bc7cb2a057', 'request_id': 'req-29c5f384-6194-4765-a5ed-d639d30cfe57'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.673402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] The token cache is being invalidate because project 8548a147f75848ffbfb9c0bc7cb2a057 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:07.674157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.674864 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29c5f384-6194-4765-a5ed-d639d30cfe57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.677126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 769/1537] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:07 2022] DELETE /identity/v3/projects/8548a147f75848ffbfb9c0bc7cb2a057 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:07.678577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:07.678947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:07.679300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:07.705738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:07.716498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:07.718821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:07.726052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0542260e-b532-4ac2-940d-8475acd0eca4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:07.730844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:07.731555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:07.731906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:07.732238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] PATH_INFO: `/v3/projects/a1b8e40724234498815a2dc703546e05` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:07.733509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1b8e40724234498815a2dc703546e05)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:07.735611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:07.735832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 770/1538] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:07.745104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:07.747255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:07.759655 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd20a38-d978-41b3-8184-88033e90bb94 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:07.764862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:07.765502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:07.765775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:07.766045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] PATH_INFO: `/v3/users/ee2133428d24465fb7e40da2c1989b4d` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:07.767480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee2133428d24465fb7e40da2c1989b4d)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:07.769132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:07.806762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1b8e40724234498815a2dc703546e05', 'request_id': 'req-22154f1e-8269-4754-b9d4-3dcc0cbbc338'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.814009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1b8e40724234498815a2dc703546e05', 'request_id': 'req-22154f1e-8269-4754-b9d4-3dcc0cbbc338'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.823546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1b8e40724234498815a2dc703546e05', 'request_id': 'req-22154f1e-8269-4754-b9d4-3dcc0cbbc338'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.841758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] The token cache is being invalidate because project a1b8e40724234498815a2dc703546e05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:07.842876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.843827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.849746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 769/1539] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:07 2022] DELETE /identity/v3/projects/a1b8e40724234498815a2dc703546e05 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:07.858013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:07.860297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:07.892322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee2133428d24465fb7e40da2c1989b4d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.912158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee2133428d24465fb7e40da2c1989b4d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.914114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee2133428d24465fb7e40da2c1989b4d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.929647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee2133428d24465fb7e40da2c1989b4d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.930733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d8263e8-1a49-4326-a0a7-01abb4778c23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee2133428d24465fb7e40da2c1989b4d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:07.956727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 771/1540] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:07 2022] DELETE /identity/v3/users/ee2133428d24465fb7e40da2c1989b4d => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:07.958464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298ccd30-325c-42d4-922b-41144b533e20 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:07.958807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298ccd30-325c-42d4-922b-41144b533e20 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:07.959139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298ccd30-325c-42d4-922b-41144b533e20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:07.983315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-298ccd30-325c-42d4-922b-41144b533e20 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:07.985001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-298ccd30-325c-42d4-922b-41144b533e20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.009329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22154f1e-8269-4754-b9d4-3dcc0cbbc338 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.137017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 772/1541] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:08.143607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.145351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.145779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.146196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.147460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.147968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.148337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.149229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.181675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.184397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c6775f-f1ab-4c1c-87d0-83769ceb21cb service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.216106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 770/1542] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:08.270564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43326e62-a91b-4c5d-a8d2-55fcf51c178e None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.271110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43326e62-a91b-4c5d-a8d2-55fcf51c178e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.271561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43326e62-a91b-4c5d-a8d2-55fcf51c178e None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.273249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 771/1543] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:30:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:30:08.293590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.294127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.294675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.327092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:08.328910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.369946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:08.371097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.391560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:08.401617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 773/1544] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:08.402798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.413304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b6120d-074f-4cbb-a63a-52024b2a7c27 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.420087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.420725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.421173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.421316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.421973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:08.422270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.424570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.449429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 774/1546] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:08.449648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 772/1546] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:30:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:08.450347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.462565 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3065d0-8af0-4725-b29f-0c8a68b493bb admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.478262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.479432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.479706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.479966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.480796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.483209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.492989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.494534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.510555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95051fdc-446e-43ab-972f-0ac0012bc50d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.517656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.518054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.518511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.518746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:08.519209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.532124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.536100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.541572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 775/1547] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:08 2022] POST /identity/v3/projects => generated 385 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:08.542872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.549659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 773/1548] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:08.629972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cde8acd-164e-47bc-9b55-a89016ad4692 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.641910 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.642719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.643258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.643893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.645262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.649378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.683459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 776/1549] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:08 2022] POST /identity/v3/users => generated 293 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:08.684996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.699974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0391096d-e237-4642-8291-cd6d2bc11a89 service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.707681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.708814 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.709285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.709752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.710533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.712258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.728115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 774/1550] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:08 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:08.731953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.742832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bc3c8-5cb3-4aaa-9343-6d2c133b1696 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.747315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8a123-3658-4ae5-b9c0-56d0ccaab9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.747908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8a123-3658-4ae5-b9c0-56d0ccaab9cf admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.748181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8a123-3658-4ae5-b9c0-56d0ccaab9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.748443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8a123-3658-4ae5-b9c0-56d0ccaab9cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.749108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8a123-3658-4ae5-b9c0-56d0ccaab9cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.750688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8a123-3658-4ae5-b9c0-56d0ccaab9cf admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.761915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 777/1551] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:08.766043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.776195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20000450-1203-46bd-b8cb-d7eea9661e3b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:08.781165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:08.781820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.782093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.782379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] PATH_INFO: `/v3/projects/1050a13993774a2fa9a86e0865592674/users/557cb4465b0742aba602d2f55a96f5d4/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.797192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=1050a13993774a2fa9a86e0865592674, user_id=557cb4465b0742aba602d2f55a96f5d4, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:08.799870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:08.815379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 775/1552] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:08 2022] PUT /identity/v3/projects/1050a13993774a2fa9a86e0865592674/users/557cb4465b0742aba602d2f55a96f5d4/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:08.827079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:08.827366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:08.827752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:08.857300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] MFA Rules not processed for user `557cb4465b0742aba602d2f55a96f5d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:08.858511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.883398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.885664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:08.940519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 778/1553] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:08.945484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:08.999897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9b7837-c4a6-445b-8ace-f359a3abf9ef admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.011590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.012449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.013013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.013872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] PATH_INFO: `/v3/projects/1f94465a7c7d402b8108eb06f16d6a83` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.015758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f94465a7c7d402b8108eb06f16d6a83)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.026577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.046673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b6a4db-6a7b-4502-bd15-85bacbc8f7cc None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.052690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.053705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.053980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.054244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.055526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.057723 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.068467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:09.080159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f94465a7c7d402b8108eb06f16d6a83', 'request_id': 'req-b65df1ee-d9a6-4f7b-821d-6f44e186607b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.088285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f94465a7c7d402b8108eb06f16d6a83', 'request_id': 'req-b65df1ee-d9a6-4f7b-821d-6f44e186607b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.092760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 779/1554] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:08 2022] POST /identity/v3/projects => generated 381 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:09.094021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.095378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f94465a7c7d402b8108eb06f16d6a83', 'request_id': 'req-b65df1ee-d9a6-4f7b-821d-6f44e186607b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.098251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.133294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] The token cache is being invalidate because project 1f94465a7c7d402b8108eb06f16d6a83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:09.133507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.134449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.139580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 776/1555] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:08 2022] DELETE /identity/v3/projects/1f94465a7c7d402b8108eb06f16d6a83 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:09.139580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.139580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.154616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:09.221559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f2d83-c1d1-474d-8196-7217fe5d4ae7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.230196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.230870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.231167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.231421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] PATH_INFO: `/v3/projects/b52b118db972455c99dd8bc0daabd409` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.233533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b52b118db972455c99dd8bc0daabd409)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.236968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.257308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65df1ee-d9a6-4f7b-821d-6f44e186607b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.263567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ce99cb-6231-447e-b40e-2daa96a83312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.264298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ce99cb-6231-447e-b40e-2daa96a83312 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.264591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ce99cb-6231-447e-b40e-2daa96a83312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.264901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ce99cb-6231-447e-b40e-2daa96a83312 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.265705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ce99cb-6231-447e-b40e-2daa96a83312 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.267971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ce99cb-6231-447e-b40e-2daa96a83312 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.296076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 777/1556] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/users => generated 291 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:09.297753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.298105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.298455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.307393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b52b118db972455c99dd8bc0daabd409', 'request_id': 'req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.313961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b52b118db972455c99dd8bc0daabd409', 'request_id': 'req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.323499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b52b118db972455c99dd8bc0daabd409', 'request_id': 'req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.324649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:09.325832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.335158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] The token cache is being invalidate because project b52b118db972455c99dd8bc0daabd409 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:09.335445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.336100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.337636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 780/1557] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:09 2022] DELETE /identity/v3/projects/b52b118db972455c99dd8bc0daabd409 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:09.338420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.340756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.392114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 778/1558] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:09.392847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.394148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.406592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b6a769-81b0-4b46-99ea-9d136606f3b6 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.416733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.416733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.416733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.416733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] PATH_INFO: `/v3/users/6715dc20a448424c9240c67ef789bb00` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.417100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6715dc20a448424c9240c67ef789bb00)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.417100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.421996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848b1a7-d3a9-4189-919d-8aa0b1fcc722 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.427379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4eeb0b-4a20-41ba-b0a8-cc2032254a36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.427981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4eeb0b-4a20-41ba-b0a8-cc2032254a36 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.428252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4eeb0b-4a20-41ba-b0a8-cc2032254a36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.428518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4eeb0b-4a20-41ba-b0a8-cc2032254a36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.429194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4eeb0b-4a20-41ba-b0a8-cc2032254a36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.430924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4eeb0b-4a20-41ba-b0a8-cc2032254a36 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.442238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 781/1559] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:09 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:09.443691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.443985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.444243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.466803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:09.468083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.473464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6715dc20a448424c9240c67ef789bb00'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.477823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 782/1560] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:09.478625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.488679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6715dc20a448424c9240c67ef789bb00'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.490003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6715dc20a448424c9240c67ef789bb00'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.491759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd8406-91cd-42b2-a78a-32514be31564 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.497404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.497942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.498201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.498472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.499091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.499214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6715dc20a448424c9240c67ef789bb00'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.499883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6715dc20a448424c9240c67ef789bb00'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:09.500673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.506009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:09.518008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 779/1561] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:09 2022] DELETE /identity/v3/users/6715dc20a448424c9240c67ef789bb00 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:09.518008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.518249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 783/1562] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:09.522540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.525798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.528007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.538495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:09.553173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc03971-163f-4b45-a302-56e86d694d46 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.616640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44018451-1b1a-4acc-9dbe-48f8f648ab51 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.622601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.623612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.623792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.624012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.624679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:09.625030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.627658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.635303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36ce9bc-19ff-4212-87c1-f73efb8849ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.636425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ce9bc-19ff-4212-87c1-f73efb8849ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.637033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ce9bc-19ff-4212-87c1-f73efb8849ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.637572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ce9bc-19ff-4212-87c1-f73efb8849ae admin admin] PATH_INFO: `/v3/projects/64e033ccec5946e98256f3ba262b668f/users/91d83df4586a4c3d82bd61ad5b208878/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.642007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 780/1563] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:09.645930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.658617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36ce9bc-19ff-4212-87c1-f73efb8849ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=64e033ccec5946e98256f3ba262b668f, user_id=91d83df4586a4c3d82bd61ad5b208878, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.658997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2308924e-72b6-43a7-83a2-2642c84f11f5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.664136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36ce9bc-19ff-4212-87c1-f73efb8849ae admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.664803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.668690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.669019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.669593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.669933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.671928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.693193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 784/1564] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:09 2022] PUT /identity/v3/projects/64e033ccec5946e98256f3ba262b668f/users/91d83df4586a4c3d82bd61ad5b208878/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:09.700194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.700462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.700791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.708382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 781/1565] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:09.719096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.755736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] MFA Rules not processed for user `91d83df4586a4c3d82bd61ad5b208878`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:09.758027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:09.851117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 785/1566] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:09.853189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b227cf-6c2e-4e0d-bba1-66a9e0c11b18 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.859516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.864416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.866119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.866725 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.867236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.868664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.870372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a248093c-b8ca-4ee0-8188-a000bb09aed7 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.871354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.891598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.892605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.893109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.893748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.896250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.899255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.900304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 782/1567] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/users => generated 287 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:09.913074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.923779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7c6f98-2b3e-4971-b5fc-a87e520c2ca5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:09.929813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:09.930481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:09.930764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:09.931029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:09.931669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:09.933799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:09.934458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 786/1568] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/projects => generated 385 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:09.938930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.949991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 783/1569] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:09 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:09.951515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:09.962630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668789ec-1cfc-488f-992d-236689d993b6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.029167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.029855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.030272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.030548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.031293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.033617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.036309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:10.038869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:10.057551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 784/1570] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:09 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:10.058519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.060305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.066938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edce898-d990-4a42-801c-09f373e322c5 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.073222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.073850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.074261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.074617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.075158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.077214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.078951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f57e757-eb0b-4f2b-93cc-efb59070b855 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.086734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.087648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.087889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.088180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] PATH_INFO: `/v3/projects/169a683e8541425885ca20c9bf605569` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.089343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] RBAC: Authorizing `identity:delete_project(project_id=169a683e8541425885ca20c9bf605569)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.091751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.107540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 787/1571] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:09 2022] POST /identity/v3/users => generated 299 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:10.108725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.124321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a60c4b0-4abe-483a-9a20-1bb0b00acce3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.129927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.130617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.130888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.131153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] PATH_INFO: `/v3/projects/09ace0386a9743d88e8cf0c02853f523/users/3f1416a1b93d4a96b8981c35da853f36/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.148116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09ace0386a9743d88e8cf0c02853f523, user_id=3f1416a1b93d4a96b8981c35da853f36, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.150781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.158904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '169a683e8541425885ca20c9bf605569', 'request_id': 'req-cb002a80-9a5e-4723-ab08-29ff6b9337da'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.165757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '169a683e8541425885ca20c9bf605569', 'request_id': 'req-cb002a80-9a5e-4723-ab08-29ff6b9337da'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.166750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 788/1572] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:10 2022] PUT /identity/v3/projects/09ace0386a9743d88e8cf0c02853f523/users/3f1416a1b93d4a96b8981c35da853f36/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:10.169818 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.176402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '169a683e8541425885ca20c9bf605569', 'request_id': 'req-cb002a80-9a5e-4723-ab08-29ff6b9337da'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.196279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] The token cache is being invalidate because project 169a683e8541425885ca20c9bf605569 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:10.196664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.197642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb002a80-9a5e-4723-ab08-29ff6b9337da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.202508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 785/1573] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:10 2022] DELETE /identity/v3/projects/169a683e8541425885ca20c9bf605569 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:10.203962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.204246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.204515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.234664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] MFA Rules not processed for user `3f1416a1b93d4a96b8981c35da853f36`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:10.235889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.247636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11331557-03c5-465d-9877-a5c5d3960c39 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.254302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.254926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.255221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.255480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.256127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.257756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.268231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 789/1574] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:10 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:10.268993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.271743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.284283 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91553d46-b664-4e56-8dda-3c020c048ddc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.289717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.290453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.290700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.290969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] PATH_INFO: `/v3/users/205e96b023424da3a767d94d1732cc35` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.292205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=205e96b023424da3a767d94d1732cc35)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.294298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.308939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 786/1575] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:10.310007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.311928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.327977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0cbec-aff9-4641-a91d-dffe3ce6a293 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.335028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.335917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.336264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.336623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] PATH_INFO: `/v3/projects/dbc1be4552d44e658525f72cfb161542/users/2bec9eb8c8774e549038ef4beb4265bc/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.353238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc1be4552d44e658525f72cfb161542, user_id=2bec9eb8c8774e549038ef4beb4265bc, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.355339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '205e96b023424da3a767d94d1732cc35'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.355912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.369518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '205e96b023424da3a767d94d1732cc35'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.375943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '205e96b023424da3a767d94d1732cc35'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.378140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 787/1576] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:10 2022] PUT /identity/v3/projects/dbc1be4552d44e658525f72cfb161542/users/2bec9eb8c8774e549038ef4beb4265bc/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:10.379853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.382520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.385342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '205e96b023424da3a767d94d1732cc35'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.386102 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '205e96b023424da3a767d94d1732cc35'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.400076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 790/1577] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:10 2022] DELETE /identity/v3/users/205e96b023424da3a767d94d1732cc35 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:10.400951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.402295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.431077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:10.446431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:10.456734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe500f6-d8fb-4015-acd2-aa627009c8c7 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.486647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd81052e-6bc4-4f21-acd2-94860c4afe99 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.493257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.493460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.493701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.493973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.494618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.496226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.507396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 791/1578] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:10 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:10.512003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.520686 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.521412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.521690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.521967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.523007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697ccce-f747-405f-a1fb-33ab70d4896f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.523884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.528140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0aa4dc4-0885-4c6a-87f0-6cff969e68f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.529097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0aa4dc4-0885-4c6a-87f0-6cff969e68f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.529554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0aa4dc4-0885-4c6a-87f0-6cff969e68f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.529827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0aa4dc4-0885-4c6a-87f0-6cff969e68f2 admin admin] PATH_INFO: `/v3/projects/dbc1be4552d44e658525f72cfb161542/users/2bec9eb8c8774e549038ef4beb4265bc/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.531817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0aa4dc4-0885-4c6a-87f0-6cff969e68f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc1be4552d44e658525f72cfb161542, user_id=2bec9eb8c8774e549038ef4beb4265bc, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.534594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0aa4dc4-0885-4c6a-87f0-6cff969e68f2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.547877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 792/1579] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:10 2022] PUT /identity/v3/projects/dbc1be4552d44e658525f72cfb161542/users/2bec9eb8c8774e549038ef4beb4265bc/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:10.559285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.559572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.559898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.598822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] MFA Rules not processed for user `2bec9eb8c8774e549038ef4beb4265bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:10.600072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.603166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.604851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.616574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 788/1580] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:10.640216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.641657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.656348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84448c4-6597-430b-9144-7527c8479315 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.685793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 793/1581] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:10 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:10.694441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.696412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.725800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.726438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.726713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.726983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] PATH_INFO: `/v3/users/3f1416a1b93d4a96b8981c35da853f36` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.728177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f1416a1b93d4a96b8981c35da853f36)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.730321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.766471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d53c89-1d8a-4ab5-b24a-245aa5b28c19 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.770611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.771227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.771530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.771772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.773921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.787976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f1416a1b93d4a96b8981c35da853f36'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.796863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:10.802401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f1416a1b93d4a96b8981c35da853f36'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.803213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f1416a1b93d4a96b8981c35da853f36'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.810909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f1416a1b93d4a96b8981c35da853f36'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.811631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f1416a1b93d4a96b8981c35da853f36'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.823053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:10.828333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 789/1582] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:10 2022] DELETE /identity/v3/users/3f1416a1b93d4a96b8981c35da853f36 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:10.828333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.828333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.840669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.842132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8cdca6-5dcf-4e23-a11d-5dadb7df1622 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.851785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 794/1583] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:10 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:10.853196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.853581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.853977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.874557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:10.875759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.898380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9335fcba-10ce-4aad-9fd3-a7a757e8a94f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:10.903277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:10.903818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:10.904095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:10.904362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] PATH_INFO: `/v3/projects/d6c3e9c7558d424d9555f0a532df3588` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:10.905545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6c3e9c7558d424d9555f0a532df3588)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:10.907481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:10.959950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 795/1584] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:10.964166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6c3e9c7558d424d9555f0a532df3588', 'request_id': 'req-abb1ea75-e11d-4f48-bd2a-41cac227ecff'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.970707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:10.973022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:10.973138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6c3e9c7558d424d9555f0a532df3588', 'request_id': 'req-abb1ea75-e11d-4f48-bd2a-41cac227ecff'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:10.985113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6c3e9c7558d424d9555f0a532df3588', 'request_id': 'req-abb1ea75-e11d-4f48-bd2a-41cac227ecff'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:11.005654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] The token cache is being invalidate because project d6c3e9c7558d424d9555f0a532df3588 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:11.005947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:11.006986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-abb1ea75-e11d-4f48-bd2a-41cac227ecff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:11.013411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 790/1585] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:10 2022] DELETE /identity/v3/projects/d6c3e9c7558d424d9555f0a532df3588 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:11.014941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6330b219-b630-4b05-8d5b-4c2fd9b78275 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.015239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6330b219-b630-4b05-8d5b-4c2fd9b78275 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.015629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6330b219-b630-4b05-8d5b-4c2fd9b78275 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.047216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6330b219-b630-4b05-8d5b-4c2fd9b78275 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:11.050249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6330b219-b630-4b05-8d5b-4c2fd9b78275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:11.089245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2457434d-82ed-45cb-bb60-3b6d1fd8cb46 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.214253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 791/1586] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:11.221752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.222044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.223712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.224799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.225708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.226183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.226659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.228951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:11.256312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.258805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.267709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:11.278132 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:11.280463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 796/1587] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:11.293039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:11.334050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 792/1588] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:11.337099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:11.349883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b429d9-154c-4f7e-a651-decdcc4150b7 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.358497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.360445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.361769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.362464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.363211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:11.363329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.365345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.400853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 797/1589] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:11.405156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:11.417645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bb91e0-a10f-499e-949b-8600b736253b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.437478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.438431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.440372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.440870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.442898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.445928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.518153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 793/1590] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:11 2022] POST /identity/v3/projects => generated 373 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:11.522689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:11.543966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c821f41-a34b-4e56-a4c1-e08ca1120719 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.579126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:11.663357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.663997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.664272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.664799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.666250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.690879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.732463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 798/1591] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:11 2022] POST /identity/v3/users => generated 287 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:11.736597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:11.749542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9468a3-e0d6-48df-b7d2-1e22e6a8df84 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.755452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.756165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.757017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.757307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.758067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.759982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.778501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 794/1592] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:11 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:30:11.780935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:11.796160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7347d8e2-6740-4316-999e-0a7600b303a1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.802990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c92008-0365-4547-bfe5-4a012c9cc78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.803309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c92008-0365-4547-bfe5-4a012c9cc78f admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.803659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c92008-0365-4547-bfe5-4a012c9cc78f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.803996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c92008-0365-4547-bfe5-4a012c9cc78f admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.804891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c92008-0365-4547-bfe5-4a012c9cc78f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.806853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c92008-0365-4547-bfe5-4a012c9cc78f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.825603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 799/1593] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:11 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:11.826800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:11.855441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313fcb5-e6bc-4eba-b416-697738a4d00f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:11.862016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:11.862917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.863357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.863801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] PATH_INFO: `/v3/projects/177af31ad03c40e6b2a4a72c16451b5d/users/a2ce151a58444f84b06a6e10cdebf1c3/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.893161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] RBAC: Authorizing `identity:create_grant(project_id=177af31ad03c40e6b2a4a72c16451b5d, user_id=a2ce151a58444f84b06a6e10cdebf1c3, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:11.897554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:11.916768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 795/1594] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:11 2022] PUT /identity/v3/projects/177af31ad03c40e6b2a4a72c16451b5d/users/a2ce151a58444f84b06a6e10cdebf1c3/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:11.921664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:11.922096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:11.922623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:11.957452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:11.960332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:12.088177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 800/1595] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:12.201586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:12.203625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:12.344990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7f944-9e34-47fd-93c0-f45f98377119 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:12.357145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:12.357787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:12.358113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:12.358736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:12.380389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:12.383101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:12.400981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 796/1596] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:12.626706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:12.628149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:12.703458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe1ecbe-e84e-4899-a874-03a8f33e67fe None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:12.707859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:12.708490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:12.708773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:12.709039 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:12.711109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:12.781228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:12.782754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdce3c2-105b-4f0b-a2b6-4b90ce0de791 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:12.794807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 801/1597] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:12 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:12.883527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:12.885875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:12.911021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75351778-3d30-4414-b120-f96a7765219c service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:12.924096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:12.925508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:12.925790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:12.926152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] PATH_INFO: `/v3/projects/09ace0386a9743d88e8cf0c02853f523` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:12.931449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] RBAC: Authorizing `identity:delete_project(project_id=09ace0386a9743d88e8cf0c02853f523)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:12.933833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.048124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09ace0386a9743d88e8cf0c02853f523', 'request_id': 'req-05c9b4e2-1176-4554-97da-6fdd77cb742d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:13.060486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09ace0386a9743d88e8cf0c02853f523', 'request_id': 'req-05c9b4e2-1176-4554-97da-6fdd77cb742d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:13.075138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09ace0386a9743d88e8cf0c02853f523', 'request_id': 'req-05c9b4e2-1176-4554-97da-6fdd77cb742d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:13.102651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] The token cache is being invalidate because project 09ace0386a9743d88e8cf0c02853f523 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:13.103654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:13.105277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-05c9b4e2-1176-4554-97da-6fdd77cb742d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:13.110422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 797/1598] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:12 2022] DELETE /identity/v3/projects/09ace0386a9743d88e8cf0c02853f523 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:13.118523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.118816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.119089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.137880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:13.171806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:13.175145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:13.289574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 802/1599] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:13.298705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.299005 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.300046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.351751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:13.351751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:13.374856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 798/1600] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:13.380162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.393465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bd64dc-2784-41ce-87ea-280e51d0931d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.400569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:13.401252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.401570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.401848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.402508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:13.402861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:13.404591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.425072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.433176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 803/1601] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:13.437979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51227c0d-29ee-4a15-b207-46592524ac16 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.445025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:13.445648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.445930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.446284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.447397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:13.449879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.480360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:13.498095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 799/1602] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:13 2022] POST /identity/v3/projects => generated 375 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:13.500674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.514415 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2506514-134e-4b1a-ab6d-c20b16b1393a admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.596096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:13.596744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.597722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.597925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.598545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:13.602019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.638086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 804/1603] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:13 2022] POST /identity/v3/users => generated 288 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:13.638674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.656259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a700de-efe1-44a2-9110-1839330cde42 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.678314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:13.679054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.679434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.679893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.682078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:13.685188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.714770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.715306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 800/1604] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:13 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:30:13.727300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973094b8-0ef0-44b3-9279-5cfd4e5adbd9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.753077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:13.754405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.754681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.754947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.755587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:13.757398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.789928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 805/1605] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:13 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:13.794253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.801234 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:13.803878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:13.823912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6f1ec-c74f-4674-a844-a848f1bcaea0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.835275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93995513-d32a-4308-a739-89d61693a615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:13.836798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93995513-d32a-4308-a739-89d61693a615 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.837231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93995513-d32a-4308-a739-89d61693a615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.837511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93995513-d32a-4308-a739-89d61693a615 admin admin] PATH_INFO: `/v3/projects/62575bb44fa04c488804a9e076e6504f/users/74b28a595a504b929873b1a8a72d47ed/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.902807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93995513-d32a-4308-a739-89d61693a615 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62575bb44fa04c488804a9e076e6504f, user_id=74b28a595a504b929873b1a8a72d47ed, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:13.913918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93995513-d32a-4308-a739-89d61693a615 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:13.945438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 801/1606] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:13 2022] PUT /identity/v3/projects/62575bb44fa04c488804a9e076e6504f/users/74b28a595a504b929873b1a8a72d47ed/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:13.946858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:13.947143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:13.947417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:13.955344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54dc733-30f4-472b-8ed6-f7a04edb7458 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:13.983612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] MFA Rules not processed for user `a2ce151a58444f84b06a6e10cdebf1c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:13.984927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:13.987562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:14.020939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.021686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.022054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.022392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.024334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:14.108653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.110351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34165155-ff18-40b9-89bb-6f824a4da7ff service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.132046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 806/1607] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:13 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:14.133971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09e264-1f83-42ec-8947-330750097662 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.134225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09e264-1f83-42ec-8947-330750097662 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.134692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09e264-1f83-42ec-8947-330750097662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.169683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ac09e264-1f83-42ec-8947-330750097662 None None] MFA Rules not processed for user `74b28a595a504b929873b1a8a72d47ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:14.171540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ac09e264-1f83-42ec-8947-330750097662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:14.284577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 802/1608] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:14.298607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.300901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 807/1609] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:14.305644 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09e264-1f83-42ec-8947-330750097662 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.325111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60079a22-e427-4dfe-b6a3-0868f8d6eb8a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.330055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.330906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.331321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.331756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.342679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.344319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.357703 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 803/1610] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:14.404102 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09e264-1f83-42ec-8947-330750097662 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.414533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.415636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.415987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.416488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.417228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.420837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.438118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:14.463386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 808/1611] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:14 2022] POST /identity/v3/projects => generated 371 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:14.467266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.478058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5e42d2-a1d1-43d4-bee6-ac4ea9b35c65 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.548339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.548989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.549259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.549518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.550159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.552622 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.578689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 804/1612] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:14 2022] POST /identity/v3/users => generated 292 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:14.583162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.593846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01667c96-c199-4e37-b1f5-895a627b7294 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.601004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.601768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.602557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.602557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.603101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.604796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.617728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 809/1613] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:14 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:14.622361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.636546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48baeaf-27ac-4f44-84b6-aa67f27a2f31 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.642776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.643491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.643764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.644601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] PATH_INFO: `/v3/projects/f92c205289044f09b090c963a0ad3e16/users/ed178d64526145fdb246ab8bb101ed49/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.662125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f92c205289044f09b090c963a0ad3e16, user_id=ed178d64526145fdb246ab8bb101ed49, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.664774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.716212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 805/1614] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:14 2022] PUT /identity/v3/projects/f92c205289044f09b090c963a0ad3e16/users/ed178d64526145fdb246ab8bb101ed49/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:14.717244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.728489 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74583e93-02fa-43c2-89c8-52d05bc7cec3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.819532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d19238a-5fed-4bf6-b8b1-53154804f3b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.820298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19238a-5fed-4bf6-b8b1-53154804f3b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.820643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19238a-5fed-4bf6-b8b1-53154804f3b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.820970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19238a-5fed-4bf6-b8b1-53154804f3b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.822105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d19238a-5fed-4bf6-b8b1-53154804f3b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.825531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d19238a-5fed-4bf6-b8b1-53154804f3b0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.844849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:14.846914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 810/1615] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:14 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:14.857784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58460caf-d60d-4b0f-bfed-20dcc38ea350 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:14.876513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:14.877784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.878206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.879166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] PATH_INFO: `/v3/projects/f92c205289044f09b090c963a0ad3e16/users/ed178d64526145fdb246ab8bb101ed49/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.881247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f92c205289044f09b090c963a0ad3e16, user_id=ed178d64526145fdb246ab8bb101ed49, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:14.884683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:14.886425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:14.911657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 806/1616] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:14 2022] PUT /identity/v3/projects/f92c205289044f09b090c963a0ad3e16/users/ed178d64526145fdb246ab8bb101ed49/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:14.922779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:14.923137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:14.923324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:14.954502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] MFA Rules not processed for user `ed178d64526145fdb246ab8bb101ed49`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:14.955913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:15.056508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 811/1617] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:14 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:15.077720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:15.190050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a75e7-4741-4596-8030-ea9115468170 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:15.196252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0480dc8-9d66-4f28-90d5-fa8665bb1441 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:15.196252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0480dc8-9d66-4f28-90d5-fa8665bb1441 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:15.196252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0480dc8-9d66-4f28-90d5-fa8665bb1441 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:15.196643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0480dc8-9d66-4f28-90d5-fa8665bb1441 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:15.295834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0480dc8-9d66-4f28-90d5-fa8665bb1441 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:15.297802 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0480dc8-9d66-4f28-90d5-fa8665bb1441 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:15.310965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 807/1618] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:15.483724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:15.485327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:15.635669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84701ced-8194-48f0-a115-d3587adf9746 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:15.646840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:15.647720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:15.647994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:15.648262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] PATH_INFO: `/v3/users/5030d50b94a34b9d8958e080cf1c9c7c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:15.649648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5030d50b94a34b9d8958e080cf1c9c7c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:15.651975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:15.655201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:15.710177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5030d50b94a34b9d8958e080cf1c9c7c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:15.725748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5030d50b94a34b9d8958e080cf1c9c7c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:15.726653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5030d50b94a34b9d8958e080cf1c9c7c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:15.737156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5030d50b94a34b9d8958e080cf1c9c7c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:15.737946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e9cfff98-624f-45e2-8c09-4a715cc09488 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5030d50b94a34b9d8958e080cf1c9c7c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:15.769928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 812/1619] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:15 2022] DELETE /identity/v3/users/5030d50b94a34b9d8958e080cf1c9c7c => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:15.778652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:15.778925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:15.779178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:15.812327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:15.813617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:15.899334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:15.899834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:15.900146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:15.968173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] MFA Rules not processed for user `a2ce151a58444f84b06a6e10cdebf1c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:15.972833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:15.998030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 808/1620] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:16.064692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 813/1621] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:16.078178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:16.083751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:16.209312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:16.213951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc27fe9-20c4-44d0-a574-27dedf6f581d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:16.231475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:16.232265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:16.232517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:16.232844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:16.265698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:16.267547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:16.294032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 809/1622] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:16.966650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:16.969735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:17.004552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736f8f50-7b1f-45b7-bb72-42a956244e4c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:17.010559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:17.011248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:17.011523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:17.011799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] PATH_INFO: `/v3/projects/90817c5c259e47f2aaebfe96f44cd58f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:17.013010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90817c5c259e47f2aaebfe96f44cd58f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:17.015097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:17.098010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90817c5c259e47f2aaebfe96f44cd58f', 'request_id': 'req-687e3214-3907-42e7-bace-c2b05a3e6352'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.121278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90817c5c259e47f2aaebfe96f44cd58f', 'request_id': 'req-687e3214-3907-42e7-bace-c2b05a3e6352'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.136654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90817c5c259e47f2aaebfe96f44cd58f', 'request_id': 'req-687e3214-3907-42e7-bace-c2b05a3e6352'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.164951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] The token cache is being invalidate because project 90817c5c259e47f2aaebfe96f44cd58f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:17.165579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.167467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.169615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 814/1623] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:16 2022] DELETE /identity/v3/projects/90817c5c259e47f2aaebfe96f44cd58f => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:17.173836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:17.193781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d604e91-8338-4a3c-acfe-bfa1ba96fa6e service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:17.396477 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:17.397162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:17.397459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:17.397778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] PATH_INFO: `/v3/users/297eb23c77cb41c58db4a232887fe0c7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:17.411853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=297eb23c77cb41c58db4a232887fe0c7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:17.414066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:17.524199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '297eb23c77cb41c58db4a232887fe0c7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.552249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '297eb23c77cb41c58db4a232887fe0c7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.553984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '297eb23c77cb41c58db4a232887fe0c7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.570357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '297eb23c77cb41c58db4a232887fe0c7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.572232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '297eb23c77cb41c58db4a232887fe0c7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:17.617152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 810/1624] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:17 2022] DELETE /identity/v3/users/297eb23c77cb41c58db4a232887fe0c7 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:18.194481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.196349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.213926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e3214-3907-42e7-bace-c2b05a3e6352 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.221983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:18.314014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.314741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.315083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.315454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] PATH_INFO: `/v3/projects/74536e0d3013450aa31a6eba8afbdd8b` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.317305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74536e0d3013450aa31a6eba8afbdd8b)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.319352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.348969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.355346 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.370914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518f760c-bac5-436b-b3fb-b318dabcd1d1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.382323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74536e0d3013450aa31a6eba8afbdd8b', 'request_id': 'req-84b876b6-fbc5-4b91-8065-aaf900187777'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.388929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74536e0d3013450aa31a6eba8afbdd8b', 'request_id': 'req-84b876b6-fbc5-4b91-8065-aaf900187777'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.397480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74536e0d3013450aa31a6eba8afbdd8b', 'request_id': 'req-84b876b6-fbc5-4b91-8065-aaf900187777'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.412557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] The token cache is being invalidate because project 74536e0d3013450aa31a6eba8afbdd8b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:18.412785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.413456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-84b876b6-fbc5-4b91-8065-aaf900187777 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.415038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 815/1625] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:18 2022] DELETE /identity/v3/projects/74536e0d3013450aa31a6eba8afbdd8b => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:18.420044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd47d90-6279-4a4a-9064-9db9bf79a139 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.420323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd47d90-6279-4a4a-9064-9db9bf79a139 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.420595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd47d90-6279-4a4a-9064-9db9bf79a139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.434964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.435599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.435904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.436172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] PATH_INFO: `/v3/users/74b28a595a504b929873b1a8a72d47ed` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.437305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=74b28a595a504b929873b1a8a72d47ed)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.439218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.440037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6cd47d90-6279-4a4a-9064-9db9bf79a139 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:18.441331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6cd47d90-6279-4a4a-9064-9db9bf79a139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.455926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 816/1626] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:18.456131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:18.458744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.459159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.459524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.482052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:18.483317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.489968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74b28a595a504b929873b1a8a72d47ed'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.492722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 817/1627] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:18.496836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.503564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74b28a595a504b929873b1a8a72d47ed'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.504470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74b28a595a504b929873b1a8a72d47ed'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.506158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fbece8-b81e-4940-b98d-06872ffa2976 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.511264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.511852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.512125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.512246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74b28a595a504b929873b1a8a72d47ed'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.512467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.512785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6b4d14cb-6df0-4ca1-9ae2-40650b8918cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74b28a595a504b929873b1a8a72d47ed'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.513093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:18.513394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.515031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.526493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 811/1628] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:18 2022] DELETE /identity/v3/users/74b28a595a504b929873b1a8a72d47ed => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:18.527812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 818/1629] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:30:18.531500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.531821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.531945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.532288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.533523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.546758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eea4676-7322-4b9f-9ea6-85c4b60bcfa1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.552458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:18.553700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.604402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.605198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.605630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.606108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.607182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.609184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.612400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 812/1630] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:18.629723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.631414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:18.632783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.644861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 819/1631] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:18 2022] POST /identity/v3/projects => generated 365 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:18.645966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.701845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc89508-a488-4be4-ae84-9943a5f7aa40 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.725841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5a7411-34e9-4017-b641-a337f76bdf63 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.754202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.754394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.754675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.754943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.755609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.762602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.808263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 820/1632] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:18 2022] POST /identity/v3/users => generated 283 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:18.809356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.811381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:18.821032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:18.826332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e8bac7-f79f-476e-90e2-2cc088a31005 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.829583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.830556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.831228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.832297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.832980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.833676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.834506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.834797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] PATH_INFO: `/v3/users/557cb4465b0742aba602d2f55a96f5d4` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.836894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] RBAC: Authorizing `identity:delete_user(user_id=557cb4465b0742aba602d2f55a96f5d4)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.841450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.849692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.852041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.871582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 813/1633] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:18.872611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.883118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4afd5c6-cdb1-4089-a756-0c02a4b05402 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.891778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.893065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.894688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.895592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.897288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.901617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.914387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '557cb4465b0742aba602d2f55a96f5d4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.924861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 814/1634] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:18 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:18.929337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:18.945816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '557cb4465b0742aba602d2f55a96f5d4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.946516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '557cb4465b0742aba602d2f55a96f5d4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.950584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ccead3-9cae-4dec-b806-601bd43fb402 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:18.958294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:18.958626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '557cb4465b0742aba602d2f55a96f5d4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.959402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-af31458e-350a-4b6e-a1ff-14a95349334a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '557cb4465b0742aba602d2f55a96f5d4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:18.967935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.972844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.973131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.973896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:18.975503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:18.978776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 821/1635] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:18 2022] DELETE /identity/v3/users/557cb4465b0742aba602d2f55a96f5d4 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:18.993112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 815/1636] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:18 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:18.995986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:18.996286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:18.996753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:18.997735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.001609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:19.017118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c5c16c-013c-486f-a60e-2d1a53388473 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.047561 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:19.051779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:19.095272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d25e7a-ce44-47fd-83ac-be488a259eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.096535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d25e7a-ce44-47fd-83ac-be488a259eed admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.096843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d25e7a-ce44-47fd-83ac-be488a259eed admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.097664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d25e7a-ce44-47fd-83ac-be488a259eed admin admin] PATH_INFO: `/v3/projects/2059b4fceaea4b33acb41ced24a4acab/users/5b1fc39f8dbe41d9b9feae1765ee95f3/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.153726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d25e7a-ce44-47fd-83ac-be488a259eed admin admin] RBAC: Authorizing `identity:create_grant(project_id=2059b4fceaea4b33acb41ced24a4acab, user_id=5b1fc39f8dbe41d9b9feae1765ee95f3, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.156439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d25e7a-ce44-47fd-83ac-be488a259eed admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.188832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 816/1637] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:18 2022] PUT /identity/v3/projects/2059b4fceaea4b33acb41ced24a4acab/users/5b1fc39f8dbe41d9b9feae1765ee95f3/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:19.200449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.200932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.201219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.238634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:19.256339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] MFA Rules not processed for user `5b1fc39f8dbe41d9b9feae1765ee95f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:19.257872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:19.272031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 822/1638] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:19.286657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.290403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:19.365859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 817/1639] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:19 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:19.368835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.408571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de86bc8a-c861-467e-a911-d32ab4c7ea62 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.444163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:19.496953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d74d13-c921-45d3-be96-bf54aceffc1d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.499241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.499870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.500144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.500409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.505721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.506945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.508468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.508908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.511101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.522931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.531253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.532770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.553098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 823/1640] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:19.577746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 818/1641] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:19 2022] POST /identity/v3/projects => generated 365 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:19.579559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.590065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.591739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83dec21-e5bc-486c-8856-c272417e4839 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.592169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:19.679263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.679892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.680172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.680999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.681416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.683344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.714500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4924be-56a5-4481-aacc-a6b2b0b3b587 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.715361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 824/1642] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:19 2022] POST /identity/v3/users => generated 289 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:19.716684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.718995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.720200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.720593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.720992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] PATH_INFO: `/v3/projects/62575bb44fa04c488804a9e076e6504f` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.722711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] RBAC: Authorizing `identity:delete_project(project_id=62575bb44fa04c488804a9e076e6504f)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.724516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.732245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1131614-ef67-41cf-b6c6-b20726797823 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.740063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.740835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.741410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.741695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.742467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.744228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.770554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 825/1643] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:19 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:19.774476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.784921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7564a-aa64-4d34-bb88-b413532ab510 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.790201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.790881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.791178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.791436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] PATH_INFO: `/v3/projects/bb709f29eb544fdfb03e721d9ca6f2ec/users/90bf099abb374f398c090bd3510a17f7/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.811368 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb709f29eb544fdfb03e721d9ca6f2ec, user_id=90bf099abb374f398c090bd3510a17f7, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.814081 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:19.814821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62575bb44fa04c488804a9e076e6504f', 'request_id': 'req-606807f6-bc69-4404-946e-138f99d4565a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:19.822305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62575bb44fa04c488804a9e076e6504f', 'request_id': 'req-606807f6-bc69-4404-946e-138f99d4565a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:19.830155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62575bb44fa04c488804a9e076e6504f', 'request_id': 'req-606807f6-bc69-4404-946e-138f99d4565a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:19.836789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 826/1644] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:19 2022] PUT /identity/v3/projects/bb709f29eb544fdfb03e721d9ca6f2ec/users/90bf099abb374f398c090bd3510a17f7/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:19.839211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.851015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] The token cache is being invalidate because project 62575bb44fa04c488804a9e076e6504f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:19.851447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:19.852275 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:19.854300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 819/1645] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:19 2022] DELETE /identity/v3/projects/62575bb44fa04c488804a9e076e6504f => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:19.854530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:19.859338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:19.861734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:19.949513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423b9d5f-cef1-4e38-98bb-af48b86d0d58 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.963025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:19.973345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606807f6-bc69-4404-946e-138f99d4565a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:19.977550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:19.978304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:19.978869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:19.979237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:19.979997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:19.990837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:20.000111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 827/1646] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:19 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:20.002725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:20.004609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:20.014266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:20.015451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:20.015779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:20.016042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] PATH_INFO: `/v3/users/ed178d64526145fdb246ab8bb101ed49` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:20.017309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed178d64526145fdb246ab8bb101ed49)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:20.030532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba73fb3-b8a0-448b-b0e4-f51a995f7034 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:20.031695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:20.044728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:20.046001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:20.046589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:20.047062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] PATH_INFO: `/v3/projects/bb709f29eb544fdfb03e721d9ca6f2ec/users/90bf099abb374f398c090bd3510a17f7/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:20.051231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb709f29eb544fdfb03e721d9ca6f2ec, user_id=90bf099abb374f398c090bd3510a17f7, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:20.054407 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:20.080814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 828/1647] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:20 2022] PUT /identity/v3/projects/bb709f29eb544fdfb03e721d9ca6f2ec/users/90bf099abb374f398c090bd3510a17f7/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:20.080814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:20.090222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:20.155995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed178d64526145fdb246ab8bb101ed49'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.238721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed178d64526145fdb246ab8bb101ed49'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.240805 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed178d64526145fdb246ab8bb101ed49'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.256545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed178d64526145fdb246ab8bb101ed49'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.257876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f6bcdf1b-7ad6-4a10-8e01-ab5b89562db5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed178d64526145fdb246ab8bb101ed49'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.276653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24624a82-5dca-4002-a8f1-77d80154dde7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:20.290943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:20.291700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:20.292988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:20.293395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] PATH_INFO: `/v3/projects/1050a13993774a2fa9a86e0865592674` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:20.295466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1050a13993774a2fa9a86e0865592674)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:20.299098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:20.315367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 820/1648] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:19 2022] DELETE /identity/v3/users/ed178d64526145fdb246ab8bb101ed49 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:20.318909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:20.319884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:20.320690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:20.421683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1050a13993774a2fa9a86e0865592674', 'request_id': 'req-38855116-fa62-43b2-9cae-e8ce0d0c678d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.456791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1050a13993774a2fa9a86e0865592674', 'request_id': 'req-38855116-fa62-43b2-9cae-e8ce0d0c678d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.459002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] MFA Rules not processed for user `90bf099abb374f398c090bd3510a17f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:20.460923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:20.520528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1050a13993774a2fa9a86e0865592674', 'request_id': 'req-38855116-fa62-43b2-9cae-e8ce0d0c678d'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.575031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] The token cache is being invalidate because project 1050a13993774a2fa9a86e0865592674 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:20.576328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.579311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.589541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 829/1649] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:20 2022] DELETE /identity/v3/projects/1050a13993774a2fa9a86e0865592674 => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:20.596569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:20.608895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:20.719949 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:20.811776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 821/1650] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:20 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 496 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:20.825419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:20.827719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:20.870317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38855116-fa62-43b2-9cae-e8ce0d0c678d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:20.886642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:20.887297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:20.887573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:20.887844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] PATH_INFO: `/v3/users/91d83df4586a4c3d82bd61ad5b208878` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:20.890387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91d83df4586a4c3d82bd61ad5b208878)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:20.892513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:20.952457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c5aef4-5f66-46e1-bc43-8560599ca538 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:20.962333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:20.963274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:20.963556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:20.965748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:20.984642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91d83df4586a4c3d82bd61ad5b208878'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:20.994704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:20.997498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:21.013913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91d83df4586a4c3d82bd61ad5b208878'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.015104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 822/1651] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:20 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:21.015386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91d83df4586a4c3d82bd61ad5b208878'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.021213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:21.023241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:21.044381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91d83df4586a4c3d82bd61ad5b208878'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.045292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f1246df1-2534-4884-ae66-1d5a0f17bbf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91d83df4586a4c3d82bd61ad5b208878'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.053563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:21.070367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 830/1652] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:20 2022] DELETE /identity/v3/users/91d83df4586a4c3d82bd61ad5b208878 => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:21.081001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848a4a68-ee7c-472d-bd53-f8b5d7d1bae9 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:21.268212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:21.278774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:21.279171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:21.280581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] PATH_INFO: `/v3/projects/f92c205289044f09b090c963a0ad3e16` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:21.283072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f92c205289044f09b090c963a0ad3e16)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:21.289649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:21.502507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f92c205289044f09b090c963a0ad3e16', 'request_id': 'req-02d9be74-d39a-4999-a378-b3ed2efe6255'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.526815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f92c205289044f09b090c963a0ad3e16', 'request_id': 'req-02d9be74-d39a-4999-a378-b3ed2efe6255'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.553284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f92c205289044f09b090c963a0ad3e16', 'request_id': 'req-02d9be74-d39a-4999-a378-b3ed2efe6255'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.587980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] The token cache is being invalidate because project f92c205289044f09b090c963a0ad3e16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:21.607957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.612150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02d9be74-d39a-4999-a378-b3ed2efe6255 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:21.616727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 823/1653] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:21 2022] DELETE /identity/v3/projects/f92c205289044f09b090c963a0ad3e16 => generated 0 bytes in 596 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:21.622928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:21.623542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:21.623744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:21.661743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:21.663032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:21.910570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 831/1654] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:21.918824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:21.919134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:21.919382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:21.945605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:21.947164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:21.959492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 824/1655] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:21.960245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:21.973857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3736fce0-6ece-4da2-9fb9-41da83ee7205 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:21.991109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:21.992027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:21.992467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:21.993217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:21.994037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:21.994486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:21.996386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.012754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 832/1656] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:22.017368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.028447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10da0599-8442-49c4-a5f4-a41ec101ee18 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.039097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.039782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.040061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.040843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.041524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.044915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.127840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 825/1657] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:22 2022] POST /identity/v3/projects => generated 383 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:22.129589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.149417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf184a9f-ca1b-4f96-bbde-cbfa1588e001 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.182577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.194355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.198704 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:22.245324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.247700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.248217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.248617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.249459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.251905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.286281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 833/1658] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:22 2022] POST /identity/v3/users => generated 292 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:22.287493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.289563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.304900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b5ed4-a2a4-49ce-b993-15ccad5052bd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.308665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbdc687-2a6a-411c-b54f-76f122c5228b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.315612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.317232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.317691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.318133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] PATH_INFO: `/v3/projects/64e033ccec5946e98256f3ba262b668f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.319460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64e033ccec5946e98256f3ba262b668f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.321870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.324082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.325680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.326220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.326536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] PATH_INFO: `/v3/users/5b1fc39f8dbe41d9b9feae1765ee95f3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.327810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b1fc39f8dbe41d9b9feae1765ee95f3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.330309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.337655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:22.420265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b1fc39f8dbe41d9b9feae1765ee95f3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.425806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64e033ccec5946e98256f3ba262b668f', 'request_id': 'req-295ab301-ff36-481a-91e7-8a1638881da1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.435446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b1fc39f8dbe41d9b9feae1765ee95f3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.435562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64e033ccec5946e98256f3ba262b668f', 'request_id': 'req-295ab301-ff36-481a-91e7-8a1638881da1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.437441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b1fc39f8dbe41d9b9feae1765ee95f3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.446250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64e033ccec5946e98256f3ba262b668f', 'request_id': 'req-295ab301-ff36-481a-91e7-8a1638881da1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.450120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b1fc39f8dbe41d9b9feae1765ee95f3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.451570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b1fc39f8dbe41d9b9feae1765ee95f3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.464341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] The token cache is being invalidate because project 64e033ccec5946e98256f3ba262b668f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:22.464341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.464341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.466312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 834/1659] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:22 2022] DELETE /identity/v3/projects/64e033ccec5946e98256f3ba262b668f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:22.467546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.468987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.472484 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 826/1660] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:22 2022] DELETE /identity/v3/users/5b1fc39f8dbe41d9b9feae1765ee95f3 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:22.473555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.475375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.567011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295ab301-ff36-481a-91e7-8a1638881da1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.573595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1a8a80-37b4-430f-985d-a9cb8b09b56e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.575809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4dd3f4-0069-461d-90c0-c5b2eb3af117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.576600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4dd3f4-0069-461d-90c0-c5b2eb3af117 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.577053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4dd3f4-0069-461d-90c0-c5b2eb3af117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.577567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4dd3f4-0069-461d-90c0-c5b2eb3af117 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.578589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4dd3f4-0069-461d-90c0-c5b2eb3af117 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.579393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.579810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.580087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.580402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] PATH_INFO: `/v3/users/2bec9eb8c8774e549038ef4beb4265bc` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.582536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bec9eb8c8774e549038ef4beb4265bc)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.585037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4dd3f4-0069-461d-90c0-c5b2eb3af117 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.595556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.604900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 835/1661] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:22 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:22.607911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.608204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.608507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.642453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:22.645155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.667872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 836/1662] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:22.669562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.687131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb80b656-cddd-416a-8857-e2a933c4e344 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.696232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.697294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.697817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.698384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.699250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:22.701639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:22.717338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 837/1663] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:22 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:22.718619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.722208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.744364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bec9eb8c8774e549038ef4beb4265bc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.758649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bec9eb8c8774e549038ef4beb4265bc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.759482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bec9eb8c8774e549038ef4beb4265bc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.770016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bec9eb8c8774e549038ef4beb4265bc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.771927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bec9eb8c8774e549038ef4beb4265bc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:22.809662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:22.816740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 827/1664] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:22 2022] DELETE /identity/v3/users/2bec9eb8c8774e549038ef4beb4265bc => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:22.821682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:22.830822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:22.849670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:22.907745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25520b15-7c17-4ff6-8739-7628cc5936ed admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:22.982386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:22.983062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:22.983337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:22.984238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:22.986588 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:23.029238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bc8725-7a75-4daa-a775-4d6a073e737a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:23.034326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:23.035834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:23.036282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:23.036730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] PATH_INFO: `/v3/projects/81144c673e7b44dcb3b0f700ee628719/users/bbcc7dcbf32b4048879bbc77e0998b52/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:23.062585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:23.064026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f50989-743f-4461-a190-417d27a94154 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:23.076613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] RBAC: Authorizing `identity:create_grant(project_id=81144c673e7b44dcb3b0f700ee628719, user_id=bbcc7dcbf32b4048879bbc77e0998b52, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:23.079598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 838/1665] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:23.080616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:23.107474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 828/1666] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:22 2022] PUT /identity/v3/projects/81144c673e7b44dcb3b0f700ee628719/users/bbcc7dcbf32b4048879bbc77e0998b52/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:23.113933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:23.114297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:23.115004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:23.152728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:23.154649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:23.243126 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 839/1667] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:23.255476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:23.258326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:23.358835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345a921-21bd-48ab-8a05-2a47c7aea31e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:23.362682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:23.364652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:23.365024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:23.365352 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:23.381829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:23.385274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:23.401736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 829/1668] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:23.669255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:23.672530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:23.687166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:23.688994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:23.693218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf235b0-1c26-4cea-8242-d231e6f4d5ae None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:23.700978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:23.701956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:23.702572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:23.703124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] PATH_INFO: `/v3/projects/dbc1be4552d44e658525f72cfb161542` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:23.704799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbc1be4552d44e658525f72cfb161542)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:23.706354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fca8dcc-5de8-4197-b3f0-96d50a14f510 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:23.713074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:23.718472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:23.719256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:23.719533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:23.719796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] PATH_INFO: `/v3/projects/2059b4fceaea4b33acb41ced24a4acab` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:23.721112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2059b4fceaea4b33acb41ced24a4acab)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:23.724851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:23.803832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:23.820671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2059b4fceaea4b33acb41ced24a4acab', 'request_id': 'req-9ca87745-39f9-4351-a670-c764e158f152'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.828914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2059b4fceaea4b33acb41ced24a4acab', 'request_id': 'req-9ca87745-39f9-4351-a670-c764e158f152'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.833147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbc1be4552d44e658525f72cfb161542', 'request_id': 'req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.837631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2059b4fceaea4b33acb41ced24a4acab', 'request_id': 'req-9ca87745-39f9-4351-a670-c764e158f152'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.845035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbc1be4552d44e658525f72cfb161542', 'request_id': 'req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.855846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbc1be4552d44e658525f72cfb161542', 'request_id': 'req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.861254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] The token cache is being invalidate because project 2059b4fceaea4b33acb41ced24a4acab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:23.862405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.865125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.867559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 830/1669] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:23 2022] DELETE /identity/v3/projects/2059b4fceaea4b33acb41ced24a4acab => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:23.871507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:23.871985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] The token cache is being invalidate because project dbc1be4552d44e658525f72cfb161542 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:23.872281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.872990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a25d311-a782-4b47-b1bc-cd4c5c7757e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:23.873673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:23.878596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 840/1670] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:23 2022] DELETE /identity/v3/projects/dbc1be4552d44e658525f72cfb161542 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:23.880295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7df4018-6e8a-4369-997f-0c031351afe8 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:23.880960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7df4018-6e8a-4369-997f-0c031351afe8 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:23.881334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7df4018-6e8a-4369-997f-0c031351afe8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:23.893453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:23.903832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b7df4018-6e8a-4369-997f-0c031351afe8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:23.907262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b7df4018-6e8a-4369-997f-0c031351afe8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:24.027431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 841/1671] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:24.031294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca87745-39f9-4351-a670-c764e158f152 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.036882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.037530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.038093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.038606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.041502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.041782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] PATH_INFO: `/v3/users/90bf099abb374f398c090bd3510a17f7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.042297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.043475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=90bf099abb374f398c090bd3510a17f7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.045767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.094699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:24.096426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:24.120541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 842/1672] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:24.124940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.133826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90bf099abb374f398c090bd3510a17f7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:24.145353 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48282ca7-015f-4afb-8af5-2691aafe6abe None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.150310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.150930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.151212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.151587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.152612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:24.152927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90bf099abb374f398c090bd3510a17f7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:24.153046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.153841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90bf099abb374f398c090bd3510a17f7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:24.154567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.173566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 843/1673] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:24.180145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.183187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:24.189479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90bf099abb374f398c090bd3510a17f7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:24.190670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90bf099abb374f398c090bd3510a17f7'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:24.212475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1dab0-63c2-486c-88c1-dcb765dd29d7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.217944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.218690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.219030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.219362 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.220114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.222816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.241315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 831/1674] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:23 2022] DELETE /identity/v3/users/90bf099abb374f398c090bd3510a17f7 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:24.278261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 844/1675] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:24 2022] POST /identity/v3/projects => generated 373 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:24.282385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.384963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508e7631-e3bd-42a0-b2ed-281797d91bf9 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.437553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:24.480750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.482421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.482929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.483432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.485504 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.488846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.528495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 832/1676] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:24 2022] POST /identity/v3/users => generated 287 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:24.533137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.576791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fbc09-520a-4269-94a3-5f3d89913da0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.583460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.584292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.584597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.585373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.590963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.593137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.621794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 845/1677] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:24 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:24.625503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.652998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc33af0-28a1-4bde-a7de-0a68e52a234e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.660855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8c2903-f23a-471d-bc51-b6d80fef04a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.662235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8c2903-f23a-471d-bc51-b6d80fef04a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.662619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8c2903-f23a-471d-bc51-b6d80fef04a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.662992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8c2903-f23a-471d-bc51-b6d80fef04a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.663849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8c2903-f23a-471d-bc51-b6d80fef04a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.665795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8c2903-f23a-471d-bc51-b6d80fef04a5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.679503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 833/1678] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:24 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:24.683885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.695788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.696385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.698729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f109e73e-40cf-41b6-8f51-e79c4b8daec6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.698969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.712889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef839f0f-6815-492b-a125-6c78c29b0f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:24.713821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef839f0f-6815-492b-a125-6c78c29b0f4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.714304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef839f0f-6815-492b-a125-6c78c29b0f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.715129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef839f0f-6815-492b-a125-6c78c29b0f4b admin admin] PATH_INFO: `/v3/projects/f415639794f04d77a2d489ab6c70124b/users/cfa0f3b1dc13441aa6cdae289ba3aebe/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.743891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] MFA Rules not processed for user `bbcc7dcbf32b4048879bbc77e0998b52`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:24.745653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:24.756270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef839f0f-6815-492b-a125-6c78c29b0f4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f415639794f04d77a2d489ab6c70124b, user_id=cfa0f3b1dc13441aa6cdae289ba3aebe, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:24.762872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef839f0f-6815-492b-a125-6c78c29b0f4b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:24.798516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 846/1679] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:24 2022] PUT /identity/v3/projects/f415639794f04d77a2d489ab6c70124b/users/cfa0f3b1dc13441aa6cdae289ba3aebe/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:24.809426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:24.810545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:24.810979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:24.857428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] MFA Rules not processed for user `cfa0f3b1dc13441aa6cdae289ba3aebe`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:24.860185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:24.877216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 834/1680] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:24.884971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.886384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:24.905601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:24.957199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 847/1681] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:24.967905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd489e40-4dfc-44bb-899c-21afc0ea0fe4 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:24.969911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:24.971211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:25.041718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.043239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ba8dbc-4380-4c8f-a9f3-5d247ba50c69 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:25.044018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.044480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.045531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.054401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.056927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.057526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.057941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.069832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:25.070543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:25.077054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:25.101576 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 848/1682] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:25.120788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:25.130503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:25.146216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:25.148130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:25.161730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 835/1683] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:25.164931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:25.285755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fa3dc6-3252-4d24-8d22-6b168b2ba7f9 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:25.298831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.299907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.300337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.300739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] PATH_INFO: `/v3/projects/bb709f29eb544fdfb03e721d9ca6f2ec` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.305017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb709f29eb544fdfb03e721d9ca6f2ec)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:25.307418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:25.321562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b33c07-459d-4bfc-a933-a44e91ea1e7a service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:25.336899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.337598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.337861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.338130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] PATH_INFO: `/v3/users/cfa0f3b1dc13441aa6cdae289ba3aebe` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.341884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfa0f3b1dc13441aa6cdae289ba3aebe)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:25.347530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:25.421816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb709f29eb544fdfb03e721d9ca6f2ec', 'request_id': 'req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.432237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb709f29eb544fdfb03e721d9ca6f2ec', 'request_id': 'req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.449664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb709f29eb544fdfb03e721d9ca6f2ec', 'request_id': 'req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.487371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfa0f3b1dc13441aa6cdae289ba3aebe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.494076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:25.508217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] The token cache is being invalidate because project bb709f29eb544fdfb03e721d9ca6f2ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:25.509913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.512279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-46b8e4dd-2a23-49bf-a7f9-fca6bc840b03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.513810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfa0f3b1dc13441aa6cdae289ba3aebe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.514540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfa0f3b1dc13441aa6cdae289ba3aebe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.518652 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 849/1684] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:25 2022] DELETE /identity/v3/projects/bb709f29eb544fdfb03e721d9ca6f2ec => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:25.526481 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27daae-408d-4322-9f91-9d5c0d596401 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.527511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27daae-408d-4322-9f91-9d5c0d596401 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.527971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27daae-408d-4322-9f91-9d5c0d596401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.540870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfa0f3b1dc13441aa6cdae289ba3aebe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.541575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-033efbad-0bf2-4748-8255-ee4407223fa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfa0f3b1dc13441aa6cdae289ba3aebe'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:25.567620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 836/1685] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:25 2022] DELETE /identity/v3/users/cfa0f3b1dc13441aa6cdae289ba3aebe => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:25.572446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.572761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.573035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.611966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:25.613370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:25.616966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ac27daae-408d-4322-9f91-9d5c0d596401 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:25.621795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ac27daae-408d-4322-9f91-9d5c0d596401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:25.711726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:25.738312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 850/1686] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:25.746139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.746418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.746872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.778025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:25.779230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:25.784946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 837/1687] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:25.801914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 851/1688] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:25.805535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:25.807069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:25.813051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:25.836526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5402273c-061d-4e02-931e-1adb37a1d1b7 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:25.849290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.849930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.850188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.850453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.851098 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:25.851398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:25.853817 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:25.885169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 852/1689] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:25.895274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:25.949369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf5bdb-1424-4e20-addb-256118425256 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:25.965544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.966320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.966600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.966868 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:25.967771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:25.975844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:25.991833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a0c3c-d4db-4c20-96e2-548d555a6543 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:25.997694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:25.998322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:25.998595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:25.998862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.039979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.039979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.044450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 853/1690] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:25 2022] POST /identity/v3/projects => generated 387 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:26.048309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.055588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 838/1691] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:26.168507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2aaa89-3ad1-472f-b3fe-e264ecad0bce admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.174420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.175815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.177124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.177472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.178148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.180101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.217745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 854/1692] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:26 2022] POST /identity/v3/users => generated 294 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:26.221307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.237894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c434cf2-4d7a-4adc-ba06-926f715abe0e service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.246938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.248453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.248770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.249240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.249887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.251575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.266664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 839/1693] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:26 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:26.269299 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.287089 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234b0a6-8066-4921-b888-7ab6818d4be5 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.309285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289aa42e-dd3e-4991-bb0c-484e0b68c1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.310071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289aa42e-dd3e-4991-bb0c-484e0b68c1ca admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.310413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289aa42e-dd3e-4991-bb0c-484e0b68c1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.310875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289aa42e-dd3e-4991-bb0c-484e0b68c1ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.312267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289aa42e-dd3e-4991-bb0c-484e0b68c1ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.313986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289aa42e-dd3e-4991-bb0c-484e0b68c1ca admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.329902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 855/1694] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:26 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:30:26.334166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.349826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a670ba-a6cc-4c44-9713-7c5fab448e6b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.366491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.367176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.367451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.367718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] PATH_INFO: `/v3/projects/7615ebe9ac444da992a275de345aacaf/users/95abd694c7344c4c96667015cce9e74e/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.391911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=7615ebe9ac444da992a275de345aacaf, user_id=95abd694c7344c4c96667015cce9e74e, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.436132 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.475600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 840/1695] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:26 2022] PUT /identity/v3/projects/7615ebe9ac444da992a275de345aacaf/users/95abd694c7344c4c96667015cce9e74e/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:26.477424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.478145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.478502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.516263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] MFA Rules not processed for user `95abd694c7344c4c96667015cce9e74e`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:26.517732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:26.555355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:26.628596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 856/1696] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:26 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:26.634659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.735929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de20e13-cbc5-4faf-ae08-7fe878201a3f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.742544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.743252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.743653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.744024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.745239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.746926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.796904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 841/1697] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:26 2022] POST /identity/v3/projects => generated 385 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:26.800295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.824978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288a8b9e-fc98-4f03-a3bd-4799408e1f3a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.927277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.928463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:26.928907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.930339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:26.938072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.938385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.948273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:26.956677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:26.963399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4982c3-85c1-4ae4-bd90-67a28523044f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:26.978274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:26.978978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:26.979431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:26.979707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] PATH_INFO: `/v3/projects/f415639794f04d77a2d489ab6c70124b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:26.990436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f415639794f04d77a2d489ab6c70124b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.001824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.006492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 857/1698] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:26 2022] POST /identity/v3/users => generated 299 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:27.023543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.035921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a57a9-4324-4ec3-8f9e-4ce595a76295 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.041843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.042580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.042775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.042990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.043639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.045459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.059132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 858/1699] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:27 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:27.060244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:27.063454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.080747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0875102d-b2f2-4914-8d92-3d01ba972dd7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.120281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f415639794f04d77a2d489ab6c70124b', 'request_id': 'req-e1e433fa-23b1-4b62-9a63-d89d50932c13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:27.127143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f415639794f04d77a2d489ab6c70124b', 'request_id': 'req-e1e433fa-23b1-4b62-9a63-d89d50932c13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:27.136193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f415639794f04d77a2d489ab6c70124b', 'request_id': 'req-e1e433fa-23b1-4b62-9a63-d89d50932c13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:27.151100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] The token cache is being invalidate because project f415639794f04d77a2d489ab6c70124b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:27.151388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:27.152243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e433fa-23b1-4b62-9a63-d89d50932c13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:27.157360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 842/1700] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:26 2022] DELETE /identity/v3/projects/f415639794f04d77a2d489ab6c70124b => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:27.158834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.160032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.160311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.181195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] MFA Rules not processed for user `bbcc7dcbf32b4048879bbc77e0998b52`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:27.182399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.182558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc307a4c-ce09-4af6-85a2-1c79fdcc7d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.183202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc307a4c-ce09-4af6-85a2-1c79fdcc7d7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.183475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc307a4c-ce09-4af6-85a2-1c79fdcc7d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.183739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc307a4c-ce09-4af6-85a2-1c79fdcc7d7c admin admin] PATH_INFO: `/v3/projects/b635660a7f9646a0b22192071bbb2255/users/e630e56455694a40a4c2368586a70522/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.201728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc307a4c-ce09-4af6-85a2-1c79fdcc7d7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b635660a7f9646a0b22192071bbb2255, user_id=e630e56455694a40a4c2368586a70522, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.204390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc307a4c-ce09-4af6-85a2-1c79fdcc7d7c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.225220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 859/1701] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:27 2022] PUT /identity/v3/projects/b635660a7f9646a0b22192071bbb2255/users/e630e56455694a40a4c2368586a70522/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:27.225220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7c37aa-e034-46c6-9ff7-26bf9853c0e1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.225220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7c37aa-e034-46c6-9ff7-26bf9853c0e1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.225220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7c37aa-e034-46c6-9ff7-26bf9853c0e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.246183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cc7c37aa-e034-46c6-9ff7-26bf9853c0e1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:27.247302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cc7c37aa-e034-46c6-9ff7-26bf9853c0e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.253979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 843/1702] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:27.254961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.256312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.271149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de77c28-7d35-4e1f-b34f-7d8a0e0abf06 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.275970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:27.333715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 860/1703] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:27.341552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.342065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.342640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.347368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.347987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.348285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.348527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.349205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.350854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.361295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 844/1704] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:27 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:27.366087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.367399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:27.368614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.377100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 861/1705] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:27.378216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce19d200-977d-47db-9b66-946d0d60a7f2 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.381142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.385056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5944ca-b2e5-429e-9897-02dafaae1d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.385912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5944ca-b2e5-429e-9897-02dafaae1d5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.386336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5944ca-b2e5-429e-9897-02dafaae1d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.386768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5944ca-b2e5-429e-9897-02dafaae1d5f admin admin] PATH_INFO: `/v3/projects/b635660a7f9646a0b22192071bbb2255/users/e630e56455694a40a4c2368586a70522/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.388941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5944ca-b2e5-429e-9897-02dafaae1d5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b635660a7f9646a0b22192071bbb2255, user_id=e630e56455694a40a4c2368586a70522, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.390492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34b9254-9f2e-4502-838c-444387e4dc34 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.392520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5944ca-b2e5-429e-9897-02dafaae1d5f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.396145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.396797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.397453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.397860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.398382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:27.398685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.400420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.415627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 845/1706] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:27 2022] PUT /identity/v3/projects/b635660a7f9646a0b22192071bbb2255/users/e630e56455694a40a4c2368586a70522/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:30:27.415853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 862/1707] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:30:27.417989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.418605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.418895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.419168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.429876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3f19a5-9af1-4500-9638-71f23933b838 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.463468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] MFA Rules not processed for user `e630e56455694a40a4c2368586a70522`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:27.465080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.501401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.502036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.502396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.502938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.503741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.506087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.507144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:27.552344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 863/1708] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/projects => generated 375 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:27.553630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.567905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 846/1709] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:27.581429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.583729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.651556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5188f3ad-de2d-4268-822b-adb653e8f8f9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.656841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c32ce0-47a2-4261-ad61-f92b10a9c442 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.660521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.661433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.661855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.662266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.663571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.665686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.719935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 864/1710] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/users => generated 288 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:27.721068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.733016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47443083-4fa1-43ca-8d46-a4c6869764a7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.738993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.740451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.741134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.741827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.742866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.745113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.756732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a8093f-4c1b-430e-835f-58235fd7a733 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.757918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a8093f-4c1b-430e-835f-58235fd7a733 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.758030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a8093f-4c1b-430e-835f-58235fd7a733 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.758241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a8093f-4c1b-430e-835f-58235fd7a733 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.771269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 865/1711] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:27 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:27.774359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.794542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6eb4f81-6afc-4f6b-b9f1-055782508bc4 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.809307 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.810069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.810255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.810516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.811263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.812883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.830649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 866/1712] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:27 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:30:27.836485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.843862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:27.850191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00abc694-7d9b-48e6-b568-330e93156b85 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:27.856831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:27.858180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.858180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.858356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] PATH_INFO: `/v3/projects/01036e8f062246c2a4fd6d398911cf1b/users/e84163beca8c405bbdc3c07a15dec72b/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.869401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a8093f-4c1b-430e-835f-58235fd7a733 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.871286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a8093f-4c1b-430e-835f-58235fd7a733 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.878416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] RBAC: Authorizing `identity:create_grant(project_id=01036e8f062246c2a4fd6d398911cf1b, user_id=e84163beca8c405bbdc3c07a15dec72b, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:27.881458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:27.883697 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 847/1713] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:27 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:27.907626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 867/1714] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:27 2022] PUT /identity/v3/projects/01036e8f062246c2a4fd6d398911cf1b/users/e84163beca8c405bbdc3c07a15dec72b/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:27.918216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:27.919126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:27.919483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:27.926060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:27.959798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] MFA Rules not processed for user `e84163beca8c405bbdc3c07a15dec72b`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:27.961444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:27.989641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:28.005913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6841b254-d8eb-4626-bed5-98084830997e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:28.011063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:28.011764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:28.011897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:28.012160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] PATH_INFO: `/v3/users/95abd694c7344c4c96667015cce9e74e` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:28.013611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] RBAC: Authorizing `identity:delete_user(user_id=95abd694c7344c4c96667015cce9e74e)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:28.016152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:28.066598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 848/1715] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:27 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:28.078841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:28.080510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:28.144136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95abd694c7344c4c96667015cce9e74e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:28.168756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a7d4ac-4b81-404d-8fce-7b48d868dc35 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:28.170457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95abd694c7344c4c96667015cce9e74e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:28.171522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95abd694c7344c4c96667015cce9e74e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:28.183273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:28.185158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:28.185650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:28.186371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:28.188843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95abd694c7344c4c96667015cce9e74e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:28.189687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80219917-f3fc-4b75-a2af-90e06ca3a56a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95abd694c7344c4c96667015cce9e74e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:28.192432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:28.208210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 868/1716] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:27 2022] DELETE /identity/v3/users/95abd694c7344c4c96667015cce9e74e => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:28.212738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af9c50f-6a1e-4e62-a4bc-41a837a5fa35 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:28.213025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af9c50f-6a1e-4e62-a4bc-41a837a5fa35 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:28.213309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af9c50f-6a1e-4e62-a4bc-41a837a5fa35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:28.221360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:28.222959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:28.260372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 849/1717] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:30:28.265082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3af9c50f-6a1e-4e62-a4bc-41a837a5fa35 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:28.266341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3af9c50f-6a1e-4e62-a4bc-41a837a5fa35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:28.403184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:28.421764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 869/1718] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:28.433154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:28.435471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:28.575026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fb3b6c-b725-497f-8433-c3da9493639a service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:28.699895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:28.701446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:28.701725 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:28.703403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:28.760399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:28.762839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:28.797847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 850/1719] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:29.963663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e3718-a985-4394-a920-b398f695635e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:29.964678 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e3718-a985-4394-a920-b398f695635e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:29.965320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e3718-a985-4394-a920-b398f695635e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:30.000225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-311e3718-a985-4394-a920-b398f695635e None None] MFA Rules not processed for user `e84163beca8c405bbdc3c07a15dec72b`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:30.001967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-311e3718-a985-4394-a920-b398f695635e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:30.076849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 870/1720] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:30.286506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:30.287985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:30.301196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84c387b-9bec-48f5-b4e7-4d8106486bf4 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:30.305913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:30.306544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:30.306816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:30.307081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] PATH_INFO: `/v3/projects/7615ebe9ac444da992a275de345aacaf` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:30.308198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7615ebe9ac444da992a275de345aacaf)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:30.310195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:30.328448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:30.366371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7615ebe9ac444da992a275de345aacaf', 'request_id': 'req-f423e390-9523-442a-a2f7-e4a95e49af04'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.374884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7615ebe9ac444da992a275de345aacaf', 'request_id': 'req-f423e390-9523-442a-a2f7-e4a95e49af04'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.382597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7615ebe9ac444da992a275de345aacaf', 'request_id': 'req-f423e390-9523-442a-a2f7-e4a95e49af04'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.399175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] The token cache is being invalidate because project 7615ebe9ac444da992a275de345aacaf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:30.399175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.401577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.406547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 851/1721] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:30 2022] DELETE /identity/v3/projects/7615ebe9ac444da992a275de345aacaf => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:30.407985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e3718-a985-4394-a920-b398f695635e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:30.422463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e3718-a985-4394-a920-b398f695635e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:30.495054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:30.496887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:30.497507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:30.497973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] PATH_INFO: `/v3/users/e630e56455694a40a4c2368586a70522` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:30.499463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e630e56455694a40a4c2368586a70522)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:30.503349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:30.628356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e630e56455694a40a4c2368586a70522'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.654012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e630e56455694a40a4c2368586a70522'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.654890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e630e56455694a40a4c2368586a70522'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.666251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e630e56455694a40a4c2368586a70522'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.667667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c1b1c77-326c-4ada-97da-af83bfe95c2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e630e56455694a40a4c2368586a70522'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:30.704662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 871/1722] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:30 2022] DELETE /identity/v3/users/e630e56455694a40a4c2368586a70522 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:31.506881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:31.508439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:31.527107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f423e390-9523-442a-a2f7-e4a95e49af04 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:31.606094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:31.606995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:31.607145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:31.607410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] PATH_INFO: `/v3/projects/b635660a7f9646a0b22192071bbb2255` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:31.608551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b635660a7f9646a0b22192071bbb2255)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:31.610547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:31.674680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b635660a7f9646a0b22192071bbb2255', 'request_id': 'req-3b1b35da-7b86-4676-a5a7-fdd14a214f82'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:31.681630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b635660a7f9646a0b22192071bbb2255', 'request_id': 'req-3b1b35da-7b86-4676-a5a7-fdd14a214f82'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:31.688452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b635660a7f9646a0b22192071bbb2255', 'request_id': 'req-3b1b35da-7b86-4676-a5a7-fdd14a214f82'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:31.703928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] The token cache is being invalidate because project b635660a7f9646a0b22192071bbb2255 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:31.704225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:31.705019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3b1b35da-7b86-4676-a5a7-fdd14a214f82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:31.706937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 852/1723] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:31 2022] DELETE /identity/v3/projects/b635660a7f9646a0b22192071bbb2255 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:31.712017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:31.712497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:31.712590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:31.739946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:31.741366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:31.764758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:31.822021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 872/1724] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:31.829694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:31.830037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:31.830491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:31.853691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:31.855172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:31.867952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 853/1725] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:31.872314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:31.882291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac039a2-95f0-4204-a8e1-782aa0d2a08c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:31.887524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:31.888134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:31.888412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:31.888682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:31.889357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:31.889656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:31.891258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:31.903878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 873/1726] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:31.908093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:31.918248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c41add4-6bc8-4e40-ab1a-0dd24b287546 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:31.922938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:31.923573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:31.923848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:31.924115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:31.924821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:31.926743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:31.958820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 854/1727] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:31 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:31.963035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:31.977002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b67bd-5990-4573-831a-42535a5ff9ac admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.035610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.036229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.036508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.036808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.037443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.039361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.064343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 874/1728] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:31 2022] POST /identity/v3/users => generated 291 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:32.065637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.078309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc15249-ee2c-411a-8c35-da7c1faaf9d2 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.084474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.085204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.085527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.085798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.086449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.088074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.089557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:32.106333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.117080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 855/1729] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:32 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:32.118022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f10016-8d8d-4ef9-85f1-1743dd7cf7fc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.128973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaca791-de76-42e4-b285-3e3dc4d09be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.129598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaca791-de76-42e4-b285-3e3dc4d09be9 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.129880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaca791-de76-42e4-b285-3e3dc4d09be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.130153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaca791-de76-42e4-b285-3e3dc4d09be9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.130797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaca791-de76-42e4-b285-3e3dc4d09be9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.132417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaca791-de76-42e4-b285-3e3dc4d09be9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.144133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 875/1730] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:32 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:32.148290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.158506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc12ef5-e894-48c3-ac2c-372cf7234df4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.163107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.163819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.164105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.164377 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] PATH_INFO: `/v3/projects/fac60e8056244865a1bd981b96a8aae4/users/d14fb160231a4cc58f83c49838db66b5/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.181179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fac60e8056244865a1bd981b96a8aae4, user_id=d14fb160231a4cc58f83c49838db66b5, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.184184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.198510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 856/1731] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:32 2022] PUT /identity/v3/projects/fac60e8056244865a1bd981b96a8aae4/users/d14fb160231a4cc58f83c49838db66b5/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:32.210459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.211011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.211115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.238161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] MFA Rules not processed for user `d14fb160231a4cc58f83c49838db66b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:32.239600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:32.304631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 876/1732] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:32 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:32.309133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.372520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f97d44-5c37-4bfb-8c2b-81328380e074 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.377324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.378429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.378728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.378981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.379640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.381637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.412833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 857/1733] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:32 2022] POST /identity/v3/projects => generated 381 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:30:32.415452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.426237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd654f3-aa76-4b33-8f15-46b286223587 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.438449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:32.510597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.511638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.511938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.512367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.513088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.515008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.549205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 877/1734] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:32 2022] POST /identity/v3/users => generated 297 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:32.550905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.565811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d374af0-90f3-446b-b950-7f693a8e64df admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.572546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.573711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.574165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.574589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.575596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.577687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.592168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 858/1735] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:32 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:30:32.597222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.611628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2090c1-8546-47c9-a0e3-a39ca18c2d74 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.617622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.618345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.618620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.618899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] PATH_INFO: `/v3/projects/463c69fc693c46bd84e4dbb31c243cc2/users/3417293453344cba99956647819e489a/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.637191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=463c69fc693c46bd84e4dbb31c243cc2, user_id=3417293453344cba99956647819e489a, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.639971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.671396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 878/1736] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:32 2022] PUT /identity/v3/projects/463c69fc693c46bd84e4dbb31c243cc2/users/3417293453344cba99956647819e489a/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:32.675309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.685384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8c10c8-4e41-47dc-b04e-da5796c67daa admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.776520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaacda0-62f9-4816-beee-f1e39b7529a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.777244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaacda0-62f9-4816-beee-f1e39b7529a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.777524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaacda0-62f9-4816-beee-f1e39b7529a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.778042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaacda0-62f9-4816-beee-f1e39b7529a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.778702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaacda0-62f9-4816-beee-f1e39b7529a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.782297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaacda0-62f9-4816-beee-f1e39b7529a2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.799322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 859/1737] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:32 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:32.803776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:32.832534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa3df5-1244-4f90-a8c0-19f480442e76 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:32.842373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:32.843188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.845941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.846373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] PATH_INFO: `/v3/projects/463c69fc693c46bd84e4dbb31c243cc2/users/3417293453344cba99956647819e489a/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.852406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=463c69fc693c46bd84e4dbb31c243cc2, user_id=3417293453344cba99956647819e489a, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:32.857546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:32.875870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 879/1738] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:32 2022] PUT /identity/v3/projects/463c69fc693c46bd84e4dbb31c243cc2/users/3417293453344cba99956647819e489a/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:32.886587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:32.888241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:32.889890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:32.954083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] MFA Rules not processed for user `3417293453344cba99956647819e489a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:32.955568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:33.012933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:33.081893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 860/1739] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:32 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:33.091348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:33.093167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:33.173877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f914a-b21f-4033-8f7d-336d5a4db6d8 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:33.205449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:33.212430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:33.213108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:33.213572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:33.238024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:33.241164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:33.264116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 880/1740] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:33 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:33.382314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:33.394664 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71420a-9f53-4a3a-a947-0c4f7709ff64 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:33.398617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03baae49-8bef-48d2-97d7-6f3bb7f8c8e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:33.399233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03baae49-8bef-48d2-97d7-6f3bb7f8c8e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:33.399512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03baae49-8bef-48d2-97d7-6f3bb7f8c8e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:33.399783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03baae49-8bef-48d2-97d7-6f3bb7f8c8e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:33.498385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03baae49-8bef-48d2-97d7-6f3bb7f8c8e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:33.499878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03baae49-8bef-48d2-97d7-6f3bb7f8c8e8 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:33.511203 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 861/1741] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:33 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:33.902320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:33.991257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a151c-da39-44bd-acc1-84416893ca77 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:33.999633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:34.000312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:34.000641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:34.001019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] PATH_INFO: `/v3/users/d14fb160231a4cc58f83c49838db66b5` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:34.005508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d14fb160231a4cc58f83c49838db66b5)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:34.007881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:34.027745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:34.081496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd14fb160231a4cc58f83c49838db66b5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:34.096828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd14fb160231a4cc58f83c49838db66b5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:34.097739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd14fb160231a4cc58f83c49838db66b5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:34.110413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd14fb160231a4cc58f83c49838db66b5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:34.111667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd14fb160231a4cc58f83c49838db66b5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:34.125458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 881/1742] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:33 2022] DELETE /identity/v3/users/d14fb160231a4cc58f83c49838db66b5 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:34.130778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:34.131367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:34.131367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:34.150590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:34.151807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:34.209042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 862/1743] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:34.218960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:34.220392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:34.277274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104aec1-9e99-4c22-a0c6-c82a6ae6c8f3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:34.280908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:34.281516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:34.281790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:34.282056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:34.292158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:34.294147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:34.304601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 882/1744] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:34.983073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:34.984853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:34.997273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a28ba4-0710-4575-83e0-4dda1e4c2119 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:35.001808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:35.002438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:35.003270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:35.003539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] PATH_INFO: `/v3/projects/fac60e8056244865a1bd981b96a8aae4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:35.004890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fac60e8056244865a1bd981b96a8aae4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:35.006680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:35.060866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fac60e8056244865a1bd981b96a8aae4', 'request_id': 'req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.067372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fac60e8056244865a1bd981b96a8aae4', 'request_id': 'req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.070442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:35.075627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fac60e8056244865a1bd981b96a8aae4', 'request_id': 'req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.091017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] The token cache is being invalidate because project fac60e8056244865a1bd981b96a8aae4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:35.091328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.092503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.094025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 863/1745] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:34 2022] DELETE /identity/v3/projects/fac60e8056244865a1bd981b96a8aae4 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:35.098751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:35.113009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48045e3-a3c0-4504-9e83-43a752543837 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:35.180344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:35.181286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:35.181769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:35.182288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] PATH_INFO: `/v3/users/3417293453344cba99956647819e489a` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:35.183706 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3417293453344cba99956647819e489a)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:35.186147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:35.237768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:35.251333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3417293453344cba99956647819e489a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.268259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3417293453344cba99956647819e489a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.269428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3417293453344cba99956647819e489a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.279831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3417293453344cba99956647819e489a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.281880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b9b68ae9-6c30-4921-82bb-187fa8181c29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3417293453344cba99956647819e489a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.297488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 883/1746] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:35 2022] DELETE /identity/v3/users/3417293453344cba99956647819e489a => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:35.687011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:35.689531 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:35.713754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf59ab-0588-4f72-8c5f-a11ec8fb1d89 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:35.779139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:35.779786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:35.780058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:35.780331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] PATH_INFO: `/v3/projects/463c69fc693c46bd84e4dbb31c243cc2` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:35.781612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=463c69fc693c46bd84e4dbb31c243cc2)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:35.783716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:35.843259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '463c69fc693c46bd84e4dbb31c243cc2', 'request_id': 'req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.850304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '463c69fc693c46bd84e4dbb31c243cc2', 'request_id': 'req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.857842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '463c69fc693c46bd84e4dbb31c243cc2', 'request_id': 'req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.875028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] The token cache is being invalidate because project 463c69fc693c46bd84e4dbb31c243cc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:35.875349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.876022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1b8e23cc-49c4-4901-b3f9-09cbf9bf8b73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:35.880903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 864/1747] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:35 2022] DELETE /identity/v3/projects/463c69fc693c46bd84e4dbb31c243cc2 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:35.882667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:35.882950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:35.883221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:35.909884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:35.911352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:35.981941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 884/1748] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:35.991385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:35.991665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:35.991936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.016825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:36.018216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:36.026288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 865/1749] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:36.031225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.042383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0bbdb0-68e1-4709-9c1b-0cc4b7ad1066 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.049099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.049745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.050028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.050943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.051607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:36.051906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.053820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.066992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 885/1750] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:36.071454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.081941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8a28f7-ec5a-464a-b40d-3da1fc220eb4 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.088434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.089078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.089350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.089889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.090993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.092143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.146997 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 866/1751] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:36 2022] POST /identity/v3/projects => generated 377 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:36.148773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.159938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a53be2e-e995-4260-94b7-ae72973bf744 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.262888 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.263810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.264291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.264781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.265981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.272050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.317967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 886/1752] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:36 2022] POST /identity/v3/users => generated 289 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:36.323313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.337635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38d1ff-64c6-4faf-859f-17737b49f7f7 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.346995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.347745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.348151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.348561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.349416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.351134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.368321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.381914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 867/1753] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:36 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:36.388049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afb16b-3261-4fd7-98a5-bec8e69084ed admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.405042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a5cfe-ecd4-4395-8384-bac1d3e9b0ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.405916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a5cfe-ecd4-4395-8384-bac1d3e9b0ff admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.406199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a5cfe-ecd4-4395-8384-bac1d3e9b0ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.406475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a5cfe-ecd4-4395-8384-bac1d3e9b0ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.407283 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a5cfe-ecd4-4395-8384-bac1d3e9b0ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.409542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a5cfe-ecd4-4395-8384-bac1d3e9b0ff admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.434036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 887/1754] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:36 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:36.438064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.454379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177c002d-9f36-4c82-8777-fa3840988bfd admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.460535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.461475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.462887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.462887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] PATH_INFO: `/v3/projects/774dea95831443bba1130d4642f807ae/users/7cee6946e3cc4d499a5b707e0c305a1a/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.487980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] RBAC: Authorizing `identity:create_grant(project_id=774dea95831443bba1130d4642f807ae, user_id=7cee6946e3cc4d499a5b707e0c305a1a, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.492286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.512761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 868/1755] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:36 2022] PUT /identity/v3/projects/774dea95831443bba1130d4642f807ae/users/7cee6946e3cc4d499a5b707e0c305a1a/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:36.521644 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.522140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.522470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.558382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] MFA Rules not processed for user `7cee6946e3cc4d499a5b707e0c305a1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:36.560055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:36.573314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:36.656861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 888/1756] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:36.671922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:36.673881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:36.712889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:36.799952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71376d6d-1e8f-4ca7-9bac-3ae0ff791531 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:36.804160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:36.804823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:36.805099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:36.805364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:36.816105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:36.818803 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:36.827877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 869/1757] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:44.941874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:44.943849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:45.016812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72c99a4-b9f5-487c-8a54-64fcdf6e3e57 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:45.022431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:45.023241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:45.023515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:45.023900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] PATH_INFO: `/v3/users/47f30e76f6894e5f9c06f71adde020e6` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:45.025435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] RBAC: Authorizing `identity:delete_user(user_id=47f30e76f6894e5f9c06f71adde020e6)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:45.027919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:45.083178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47f30e76f6894e5f9c06f71adde020e6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.102616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47f30e76f6894e5f9c06f71adde020e6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.103466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47f30e76f6894e5f9c06f71adde020e6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.121295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47f30e76f6894e5f9c06f71adde020e6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.122013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fe1f87a9-d454-4f18-8c8b-445b1ab6f9df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47f30e76f6894e5f9c06f71adde020e6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.144769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 889/1758] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:44 2022] DELETE /identity/v3/users/47f30e76f6894e5f9c06f71adde020e6 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:45.399584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:45.428005 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4668cbc-2dcf-4d0c-9050-a01331508d01 service glance] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:45.528152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:45.528852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:45.529119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:45.529563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] PATH_INFO: `/v3/projects/60e8d0fb43eb419ba1c5eb830cc89ee3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:45.530793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=60e8d0fb43eb419ba1c5eb830cc89ee3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:45.533072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:45.630097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60e8d0fb43eb419ba1c5eb830cc89ee3', 'request_id': 'req-cd598cc1-bd76-4950-9213-a8e15c56ad4b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.636152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60e8d0fb43eb419ba1c5eb830cc89ee3', 'request_id': 'req-cd598cc1-bd76-4950-9213-a8e15c56ad4b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.643610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60e8d0fb43eb419ba1c5eb830cc89ee3', 'request_id': 'req-cd598cc1-bd76-4950-9213-a8e15c56ad4b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.667788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] The token cache is being invalidate because project 60e8d0fb43eb419ba1c5eb830cc89ee3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:45.668379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.669316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cd598cc1-bd76-4950-9213-a8e15c56ad4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:45.676860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 870/1759] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:45 2022] DELETE /identity/v3/projects/60e8d0fb43eb419ba1c5eb830cc89ee3 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:45.678019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:45.678475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:45.678887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:45.708970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:45.712594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:45.765731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:45.837659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 890/1760] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:45.845304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895505d1-b323-457c-8732-44827654f105 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:45.846101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895505d1-b323-457c-8732-44827654f105 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:45.846407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895505d1-b323-457c-8732-44827654f105 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:45.867058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-895505d1-b323-457c-8732-44827654f105 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:45.868272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-895505d1-b323-457c-8732-44827654f105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:45.878514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 871/1761] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:45.883189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:45.896018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0031be56-387f-47dc-bd56-b7203ef90cd9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:45.901414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:45.902099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:45.902463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:45.902824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:45.903547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:45.903922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:45.905653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:45.922169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 891/1762] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:45.923320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895505d1-b323-457c-8732-44827654f105 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:45.933604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895505d1-b323-457c-8732-44827654f105 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:45.938699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:45.941560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:45.941560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:45.941560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:45.941560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:45.943258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:45.969082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:45.981239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 872/1763] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:45 2022] POST /identity/v3/projects => generated 385 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:45.985162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:45.996580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5186f333-18ea-4463-bd79-cae2d6121c83 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:46.076162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:46.076162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:46.076162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:46.076162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:46.076529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:46.076529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:46.104512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 892/1764] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:45 2022] POST /identity/v3/users => generated 293 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:46.108655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:46.121865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac91215-3616-4ca4-859f-d3728e1549d3 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:46.126647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:46.127318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:46.127593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:46.127854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:46.128497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:46.130137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:46.141197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 873/1765] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:46.145995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:46.155964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9462f9d-5af1-4928-b691-8b3dc5e348bd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:46.161242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57cd40c-c5d4-4cf9-8639-4e3c36a9cb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:46.161964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57cd40c-c5d4-4cf9-8639-4e3c36a9cb53 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:46.162272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57cd40c-c5d4-4cf9-8639-4e3c36a9cb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:46.162541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57cd40c-c5d4-4cf9-8639-4e3c36a9cb53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:46.163178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57cd40c-c5d4-4cf9-8639-4e3c36a9cb53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:46.164831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57cd40c-c5d4-4cf9-8639-4e3c36a9cb53 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:46.176339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 893/1766] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:46.180553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:46.190633 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dabba5-eb29-4a4d-a0b0-a5b85a2b9918 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:46.195362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:46.196030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:46.196286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:46.196559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] PATH_INFO: `/v3/projects/91317c221b5a4457b7b9b4bdb1ab91f9/users/3425fb6461cc441099d4b4df26ffafc4/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:46.214540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=91317c221b5a4457b7b9b4bdb1ab91f9, user_id=3425fb6461cc441099d4b4df26ffafc4, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:46.217433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:46.250905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 874/1767] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:46 2022] PUT /identity/v3/projects/91317c221b5a4457b7b9b4bdb1ab91f9/users/3425fb6461cc441099d4b4df26ffafc4/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:46.264198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ca67a-bd8f-4cc5-bfd0-c522a5005e12 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:46.264467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ca67a-bd8f-4cc5-bfd0-c522a5005e12 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:46.264767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ca67a-bd8f-4cc5-bfd0-c522a5005e12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:46.312256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ce6ca67a-bd8f-4cc5-bfd0-c522a5005e12 None None] MFA Rules not processed for user `3425fb6461cc441099d4b4df26ffafc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:46.313532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ce6ca67a-bd8f-4cc5-bfd0-c522a5005e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:46.548083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 894/1768] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:46.558613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:46.560438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:46.724993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227bfdb7-ec4c-462d-80e5-12cee79050ba admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:46.736035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:46.739735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:46.740010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:46.740439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:46.762083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:46.763758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:46.782265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 875/1769] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:48.357948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fbafb6-9d9e-4bda-9708-551fc193ec41 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:48.358461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fbafb6-9d9e-4bda-9708-551fc193ec41 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:48.358640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fbafb6-9d9e-4bda-9708-551fc193ec41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:48.379529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-46fbafb6-9d9e-4bda-9708-551fc193ec41 None None] MFA Rules not processed for user `a2ce151a58444f84b06a6e10cdebf1c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:48.380834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-46fbafb6-9d9e-4bda-9708-551fc193ec41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:48.403651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:48.453577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 895/1770] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:55.449517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:55.524402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65eeefc-c971-40c9-8435-c36d69df8794 service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:55.529529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:55.530301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:55.530682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:55.531074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] PATH_INFO: `/v3/users/3425fb6461cc441099d4b4df26ffafc4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:55.532318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3425fb6461cc441099d4b4df26ffafc4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:55.535009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:55.604140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3425fb6461cc441099d4b4df26ffafc4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:55.608847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:55.623924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3425fb6461cc441099d4b4df26ffafc4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:55.624579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3425fb6461cc441099d4b4df26ffafc4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:55.632748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3425fb6461cc441099d4b4df26ffafc4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:55.633319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3425fb6461cc441099d4b4df26ffafc4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:55.646248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 876/1771] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:30:55 2022] DELETE /identity/v3/users/3425fb6461cc441099d4b4df26ffafc4 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:55.653265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:55.654196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:55.654464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:55.703459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:55.710059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:55.803251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 896/1772] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:55.813072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:55.815069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:55.877906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ef800-53a7-41c3-908b-256bffc2473b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:55.881902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfae52b-451e-4956-bc03-07ab24309dae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:55.882380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfae52b-451e-4956-bc03-07ab24309dae service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:55.882654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfae52b-451e-4956-bc03-07ab24309dae service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:55.882951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfae52b-451e-4956-bc03-07ab24309dae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:55.894272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfae52b-451e-4956-bc03-07ab24309dae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:55.895992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfae52b-451e-4956-bc03-07ab24309dae service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:55.906431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 877/1773] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:56.825370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:56.827688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:56.842695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da725ca-8938-4c2a-8fb0-cb87a091b3f2 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:56.850333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:56.854414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:56.854414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:56.854414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] PATH_INFO: `/v3/projects/91317c221b5a4457b7b9b4bdb1ab91f9` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:56.855207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] RBAC: Authorizing `identity:delete_project(project_id=91317c221b5a4457b7b9b4bdb1ab91f9)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:56.858003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:56.923461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91317c221b5a4457b7b9b4bdb1ab91f9', 'request_id': 'req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:56.929375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91317c221b5a4457b7b9b4bdb1ab91f9', 'request_id': 'req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:56.940436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91317c221b5a4457b7b9b4bdb1ab91f9', 'request_id': 'req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:56.960103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] The token cache is being invalidate because project 91317c221b5a4457b7b9b4bdb1ab91f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:30:56.960393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:56.961080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5e9d9c13-1d09-4afc-b60a-1f2a7337479e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:30:56.962591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 897/1774] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:30:56 2022] DELETE /identity/v3/projects/91317c221b5a4457b7b9b4bdb1ab91f9 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:56.967710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:56.968181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:56.968277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:56.990418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:56.991636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:57.056619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 878/1775] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:57.064881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.065163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.065431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.085220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:57.086487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:57.094278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 898/1776] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:57.098691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.108899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201c1a79-d800-4118-8b1b-839b40d7b17c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.114024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.114650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.114927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.115189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.116155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:30:57.116482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.118165 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.132744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 879/1777] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:30:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:30:57.137200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.146940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3997f4-f3db-47d3-ab67-425f72967660 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.151778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.152377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.152658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.152960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.153594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.155496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.187752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 899/1778] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:30:57 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:57.189359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.199491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c578ca0-4b83-4af6-93be-df381f24c072 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.268270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.268952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.269228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.269509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.270389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.272502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.284420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:57.318976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 880/1779] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:30:57 2022] POST /identity/v3/users => generated 287 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:30:57.321286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.335485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ba7706-aa28-4a87-884a-ec37b4c13b4c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.341102 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.341722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.342003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.342274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.342922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.344558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.358006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 900/1780] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:57.359284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.370554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cabb8c-937c-48d2-b175-82b56ed7950e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.375473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d4a8ce-0288-41e9-88d5-e9ffd7ff246b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.376203 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4a8ce-0288-41e9-88d5-e9ffd7ff246b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.376548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4a8ce-0288-41e9-88d5-e9ffd7ff246b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.376860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4a8ce-0288-41e9-88d5-e9ffd7ff246b admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.377634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4a8ce-0288-41e9-88d5-e9ffd7ff246b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.379267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4a8ce-0288-41e9-88d5-e9ffd7ff246b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.391309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 881/1781] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:30:57.396303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.408160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd0795-8fb8-410d-bf1b-a34e821c9142 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.413450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ac558-da39-472e-b3b0-bd33d50a35c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.414212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ac558-da39-472e-b3b0-bd33d50a35c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.415206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ac558-da39-472e-b3b0-bd33d50a35c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.415206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ac558-da39-472e-b3b0-bd33d50a35c8 admin admin] PATH_INFO: `/v3/projects/2c97c0fb4b9d411eba5f2326ce32bf18/users/70a1d657cea84d0d8fb285021ae35a2c/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.428254 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:30:57.433517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119ac558-da39-472e-b3b0-bd33d50a35c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c97c0fb4b9d411eba5f2326ce32bf18, user_id=70a1d657cea84d0d8fb285021ae35a2c, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.438147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119ac558-da39-472e-b3b0-bd33d50a35c8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.465332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 901/1782] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:30:57 2022] PUT /identity/v3/projects/2c97c0fb4b9d411eba5f2326ce32bf18/users/70a1d657cea84d0d8fb285021ae35a2c/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:30:57.475460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.475742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.476010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.509907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] MFA Rules not processed for user `70a1d657cea84d0d8fb285021ae35a2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:57.511710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:57.527849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.528139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.528728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.554461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:57.556717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:57.586151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 882/1783] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:30:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:57.595578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.597148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:57.637045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 902/1784] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:30:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:30:57.684194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:57.685034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566a218-5999-44f7-9728-43d5cfe819d2 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.685615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:57.693440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccbcdde-8b6a-4fac-b92f-9852914ed983 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.694188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccbcdde-8b6a-4fac-b92f-9852914ed983 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.694472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccbcdde-8b6a-4fac-b92f-9852914ed983 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.694740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccbcdde-8b6a-4fac-b92f-9852914ed983 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.704604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccbcdde-8b6a-4fac-b92f-9852914ed983 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.706834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccbcdde-8b6a-4fac-b92f-9852914ed983 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.716866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 883/1785] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:30:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:30:57.786088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecab0612-9a53-4677-8d66-d8608ed61140 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:57.792802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:57.793693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:57.794430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:57.794653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:57.809434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:57.811249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:57.826105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 903/1786] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:30:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:30:58.570386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef5b550-cf93-4f80-870f-a34739039493 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:58.570864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef5b550-cf93-4f80-870f-a34739039493 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:58.570990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef5b550-cf93-4f80-870f-a34739039493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:58.590960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-aef5b550-cf93-4f80-870f-a34739039493 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:58.592331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-aef5b550-cf93-4f80-870f-a34739039493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:58.600521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 884/1787] 10.222.0.23 () {58 vars in 967 bytes} [Wed Oct 12 02:30:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:58.614715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:58.625293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553df2c8-61e2-4bec-afc9-ec2069e8e759 service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:58.630802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:58.631430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:58.631708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:58.631980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:58.641559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:58.644119 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:58.653129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 904/1788] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:30:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:30:58.884944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1560cc36-ae97-4661-b375-1fd641f9eb69 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:58.885234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1560cc36-ae97-4661-b375-1fd641f9eb69 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:58.885527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1560cc36-ae97-4661-b375-1fd641f9eb69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:58.905250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1560cc36-ae97-4661-b375-1fd641f9eb69 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:58.906565 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1560cc36-ae97-4661-b375-1fd641f9eb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:58.913836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 885/1789] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:30:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:58.926020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:58.936509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecfc3b-91fd-4f34-b603-64f1a9f4e59f service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:58.942182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:58.942791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:58.943061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:58.943326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:58.951486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:58.953071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:58.961693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 905/1790] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:30:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:30:59.328806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:59.329295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:59.329426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:59.349394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:30:59.350550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:30:59.358942 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 886/1791] 10.222.0.23 () {58 vars in 967 bytes} [Wed Oct 12 02:30:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:30:59.372313 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:30:59.383108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6dd69a-a901-469a-ace1-9ee6c3a6dfb3 service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:30:59.393471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da116c9f-a88e-4591-81e3-49e09978ac5e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:30:59.394267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da116c9f-a88e-4591-81e3-49e09978ac5e service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:30:59.394545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da116c9f-a88e-4591-81e3-49e09978ac5e service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:30:59.394810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da116c9f-a88e-4591-81e3-49e09978ac5e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:30:59.403345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da116c9f-a88e-4591-81e3-49e09978ac5e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:30:59.404942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da116c9f-a88e-4591-81e3-49e09978ac5e service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:30:59.413682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 906/1792] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:30:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:31:00.620808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:00.670551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:00.710373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02deed92-e6b0-4eeb-ad1b-af6d894ea7e4 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:00.715023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:00.715756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:00.716129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:00.716494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] PATH_INFO: `/v3/users/70a1d657cea84d0d8fb285021ae35a2c` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:00.717842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] RBAC: Authorizing `identity:delete_user(user_id=70a1d657cea84d0d8fb285021ae35a2c)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:00.720390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:00.767796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70a1d657cea84d0d8fb285021ae35a2c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:00.779779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70a1d657cea84d0d8fb285021ae35a2c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:00.780446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70a1d657cea84d0d8fb285021ae35a2c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:00.788029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70a1d657cea84d0d8fb285021ae35a2c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:00.788642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70a1d657cea84d0d8fb285021ae35a2c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:00.803324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 887/1793] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:00 2022] DELETE /identity/v3/users/70a1d657cea84d0d8fb285021ae35a2c => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:00.807780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:00.808276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:00.808366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:00.827325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:00.828540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:00.874553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:00.896153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 907/1794] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:00.908290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:00.909823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:00.972318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eeb888-ff3d-4d3f-8ca4-723b2d855359 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:00.975933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c1a781-1320-4956-ae5b-76b5390e56ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:00.976547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c1a781-1320-4956-ae5b-76b5390e56ae service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:00.976874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c1a781-1320-4956-ae5b-76b5390e56ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:00.977147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c1a781-1320-4956-ae5b-76b5390e56ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:00.987211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c1a781-1320-4956-ae5b-76b5390e56ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:00.988654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c1a781-1320-4956-ae5b-76b5390e56ae service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:00.998677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 888/1795] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:31:01.287020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.288871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.303208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec840915-dcc5-4472-9597-bec01531f54f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.310233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.310233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.310233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.310233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] PATH_INFO: `/v3/projects/2c97c0fb4b9d411eba5f2326ce32bf18` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.310872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c97c0fb4b9d411eba5f2326ce32bf18)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.313906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.363928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c97c0fb4b9d411eba5f2326ce32bf18', 'request_id': 'req-3336a029-e361-48e2-a417-a53dc34af46b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.369390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c97c0fb4b9d411eba5f2326ce32bf18', 'request_id': 'req-3336a029-e361-48e2-a417-a53dc34af46b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.376931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c97c0fb4b9d411eba5f2326ce32bf18', 'request_id': 'req-3336a029-e361-48e2-a417-a53dc34af46b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.388008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] The token cache is being invalidate because project 2c97c0fb4b9d411eba5f2326ce32bf18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:01.388292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.389001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3336a029-e361-48e2-a417-a53dc34af46b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.390539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 908/1796] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:01 2022] DELETE /identity/v3/projects/2c97c0fb4b9d411eba5f2326ce32bf18 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:01.395976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.396259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.396545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.415717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:01.416945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.465935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:01.478685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 889/1797] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:01.486510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.486724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.486990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.507110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:01.508317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.516142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 909/1798] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:01.520129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.529768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff0b5e-9ed6-44b0-a22c-611aabd54be7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.534873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.535473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.535746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.536014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.536687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:31:01.537012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.538653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.552889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 890/1799] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:31:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:31:01.557216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.567043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c942d8-6264-4323-bbe7-29b08dd1389a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.571663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.572266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.572536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.572833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.573497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.575518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.606720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 910/1800] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:31:01 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:31:01.607898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.621604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ede1ea5-e4db-440e-87cd-d7d6a7610f08 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.684598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.685414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.685705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.685980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.686939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.689354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.693699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.695947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.711372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d58e3-b714-4be7-872f-6d3e0c8005ca admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.713401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 891/1801] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:31:01 2022] POST /identity/v3/users => generated 289 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:31:01.716026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.716660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.716977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.717456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] PATH_INFO: `/v3/users/7cee6946e3cc4d499a5b707e0c305a1a` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.717659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.718778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7cee6946e3cc4d499a5b707e0c305a1a)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.720688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.728131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d9c918-4b04-4ae3-b9f3-92e4d04ae4b4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.732923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.733520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.733800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.734070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.735080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.737134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.752549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 892/1802] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:01 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:31:01.756849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.767759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc2aae-b330-4d5b-94dd-21dd773805ce admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.774427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.775151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.775435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.775898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.776373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.778205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.787140 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7cee6946e3cc4d499a5b707e0c305a1a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.789257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 893/1803] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:31:01.790164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.792824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.804472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7cee6946e3cc4d499a5b707e0c305a1a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.805369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7cee6946e3cc4d499a5b707e0c305a1a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.807286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56204ca-35d2-47c1-bd0c-04d36559f9ae admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.811964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.812578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.812900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7cee6946e3cc4d499a5b707e0c305a1a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.813467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.813467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] PATH_INFO: `/v3/users/e84163beca8c405bbdc3c07a15dec72b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.813680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7cee6946e3cc4d499a5b707e0c305a1a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.814272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] RBAC: Authorizing `identity:delete_user(user_id=e84163beca8c405bbdc3c07a15dec72b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.816194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.826107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 911/1804] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:01 2022] DELETE /identity/v3/users/7cee6946e3cc4d499a5b707e0c305a1a => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:01.827526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:01.830138 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.844886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:01.872642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e84163beca8c405bbdc3c07a15dec72b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.885024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e84163beca8c405bbdc3c07a15dec72b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.885769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e84163beca8c405bbdc3c07a15dec72b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.894516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e84163beca8c405bbdc3c07a15dec72b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.895230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-71943b18-926d-49e1-bc30-1a8aa0a3badb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e84163beca8c405bbdc3c07a15dec72b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:01.902911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f4016-81f7-4640-89ab-610e40e9f184 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:01.909092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f2b0e-8077-44e5-a03d-e3cc7a8aa25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:01.909774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f2b0e-8077-44e5-a03d-e3cc7a8aa25e admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.909869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 894/1805] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:01 2022] DELETE /identity/v3/users/e84163beca8c405bbdc3c07a15dec72b => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:01.910052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f2b0e-8077-44e5-a03d-e3cc7a8aa25e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.910378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f2b0e-8077-44e5-a03d-e3cc7a8aa25e admin admin] PATH_INFO: `/v3/projects/d1067f8f40df4805a9b9dc16a2dc7eb1/users/c7bbcf11897b45229f3eebb7d8f2e2b0/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.911113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18995ec2-4b76-491c-ae12-1d4f816685c8 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.911395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18995ec2-4b76-491c-ae12-1d4f816685c8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.911714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18995ec2-4b76-491c-ae12-1d4f816685c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.928037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f2b0e-8077-44e5-a03d-e3cc7a8aa25e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1067f8f40df4805a9b9dc16a2dc7eb1, user_id=c7bbcf11897b45229f3eebb7d8f2e2b0, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:01.930843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f2b0e-8077-44e5-a03d-e3cc7a8aa25e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:01.931274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-18995ec2-4b76-491c-ae12-1d4f816685c8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:01.932471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-18995ec2-4b76-491c-ae12-1d4f816685c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:01.944951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 912/1806] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:01 2022] PUT /identity/v3/projects/d1067f8f40df4805a9b9dc16a2dc7eb1/users/c7bbcf11897b45229f3eebb7d8f2e2b0/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:01.945961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:01.946243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:01.946536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:01.959065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:01.966453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:01.967658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.014280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 895/1807] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:02.015907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.016194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.016626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.036953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 913/1808] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:02.037942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.039405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.050823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] MFA Rules not processed for user `c7bbcf11897b45229f3eebb7d8f2e2b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:02.052043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.118351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9d2715-4f32-41f0-9098-2910642bd11f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.124317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.125100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.125377 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.125647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.137411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.139376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.147740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 896/1809] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:02.147740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.156162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1247a5-5fe0-4f9b-891b-af349c38ba0b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.160931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 914/1810] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:02.161836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.162821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.163438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.164131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.164131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.166093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.188898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e482f8-b7c4-4069-ab3c-34718a64c4bc service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.194602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.195547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.195831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.196198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.196853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.198905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.199902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.201033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.214766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 897/1811] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:02.240638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 915/1812] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:31:02 2022] POST /identity/v3/projects => generated 375 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:02.244459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.257452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5087d0e7-a5a7-4482-98f9-93691a7d3fc2 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.332823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.333457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.333735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.334006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.334657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.336881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.378924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 898/1813] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:31:02 2022] POST /identity/v3/users => generated 294 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:02.382995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.394826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667c6e0e-001a-4083-95a3-a824cbb81dbc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.403760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.405198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.405809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.406679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.408314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.410301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.421978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 916/1814] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:02.427153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.440438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3a293-bc5a-4342-b33d-b934d808403f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.445427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.446098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.446376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.446648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] PATH_INFO: `/v3/projects/422cc110086d4572a85f957f1d02505a/users/7264a0f671b344fbaee53b69d29e7613/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.463465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=422cc110086d4572a85f957f1d02505a, user_id=7264a0f671b344fbaee53b69d29e7613, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.466935 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.481941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 899/1815] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:02 2022] PUT /identity/v3/projects/422cc110086d4572a85f957f1d02505a/users/7264a0f671b344fbaee53b69d29e7613/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:02.486548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.493546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:02.498308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f71347-e72e-479b-888d-6aaef3df2997 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.553793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.555961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.564359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:02.618599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.619314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.619591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.619858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.620521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.622253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.635195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 917/1816] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:02.639697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.656612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707cad45-da02-44d3-ad40-d646586372f7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.662785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afdeef-c1d1-408e-87d7-f12814644f69 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.663427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef3d0a-88ac-4875-a522-396cf98c37fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.664144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef3d0a-88ac-4875-a522-396cf98c37fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.664509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef3d0a-88ac-4875-a522-396cf98c37fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.664759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef3d0a-88ac-4875-a522-396cf98c37fd admin admin] PATH_INFO: `/v3/projects/422cc110086d4572a85f957f1d02505a/users/7264a0f671b344fbaee53b69d29e7613/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.667936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cef3d0a-88ac-4875-a522-396cf98c37fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=422cc110086d4572a85f957f1d02505a, user_id=7264a0f671b344fbaee53b69d29e7613, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.671467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.672339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.672760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.676861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] PATH_INFO: `/v3/projects/01036e8f062246c2a4fd6d398911cf1b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.677017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cef3d0a-88ac-4875-a522-396cf98c37fd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.680072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01036e8f062246c2a4fd6d398911cf1b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.680272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.691631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 918/1817] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:02 2022] PUT /identity/v3/projects/422cc110086d4572a85f957f1d02505a/users/7264a0f671b344fbaee53b69d29e7613/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:02.702055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2c78f7-5af4-4a15-b7cb-7390296ad693 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.702358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2c78f7-5af4-4a15-b7cb-7390296ad693 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.702633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2c78f7-5af4-4a15-b7cb-7390296ad693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.742025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0f2c78f7-5af4-4a15-b7cb-7390296ad693 None None] MFA Rules not processed for user `7264a0f671b344fbaee53b69d29e7613`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:02.743323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0f2c78f7-5af4-4a15-b7cb-7390296ad693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.776637 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01036e8f062246c2a4fd6d398911cf1b', 'request_id': 'req-cc34e438-62cc-4ce4-b07a-57a9b39f3682'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:02.784432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01036e8f062246c2a4fd6d398911cf1b', 'request_id': 'req-cc34e438-62cc-4ce4-b07a-57a9b39f3682'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:02.795285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01036e8f062246c2a4fd6d398911cf1b', 'request_id': 'req-cc34e438-62cc-4ce4-b07a-57a9b39f3682'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:02.822039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] The token cache is being invalidate because project 01036e8f062246c2a4fd6d398911cf1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:02.822337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:02.823299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:02.825307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 900/1818] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:02 2022] DELETE /identity/v3/projects/01036e8f062246c2a4fd6d398911cf1b => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:02.830503 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.831913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.852315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 919/1819] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:02 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:31:02.853760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.854041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.854424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.880941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:02.882312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.912047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc34e438-62cc-4ce4-b07a-57a9b39f3682 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:02.917411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:02.918313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:02.918313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:02.918525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] PATH_INFO: `/v3/projects/774dea95831443bba1130d4642f807ae` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:02.919982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=774dea95831443bba1130d4642f807ae)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:02.921972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:02.959159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 920/1820] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:02.960211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:02.961760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:02.975717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:02.977045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:02.987310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '774dea95831443bba1130d4642f807ae', 'request_id': 'req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:02.994208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '774dea95831443bba1130d4642f807ae', 'request_id': 'req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:03.001695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '774dea95831443bba1130d4642f807ae', 'request_id': 'req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:03.014102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] The token cache is being invalidate because project 774dea95831443bba1130d4642f807ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:03.014384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:03.015008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3e22d6c1-2bde-4314-9ddc-9a7ffc716e6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:03.016511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 901/1821] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:02 2022] DELETE /identity/v3/projects/774dea95831443bba1130d4642f807ae => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:03.021319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37938f-bb88-402b-8d44-fc2750bd2cbe None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.021487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f4256-2579-4402-a32a-84bb26981459 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.021661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f4256-2579-4402-a32a-84bb26981459 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.021931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f4256-2579-4402-a32a-84bb26981459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.044433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f30f4256-2579-4402-a32a-84bb26981459 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:03.045583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f30f4256-2579-4402-a32a-84bb26981459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.073474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.074100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.074488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.074938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.076747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.111894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 902/1822] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:03.113152 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.113439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.113969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.134727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:03.136017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.143980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 903/1823] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:03.144770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.151147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.152772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44f0a7c-3b50-460a-9287-d3a48d5057f4 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.153906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1950ae20-d820-4b4c-a85c-09cbd6dd758c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.158982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.159561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.159835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.160101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.160759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:31:03.161059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.162596 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 921/1824] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:03.162712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.163832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.164269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.164540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.175825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 904/1825] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:31:03.180150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.185441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:03.186648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.190866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392abf76-2999-487d-9f51-f76784d0e16d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.195407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.195619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 922/1826] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:03.197323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.197675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.198060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.199779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.200669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.203569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.211932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27559c69-b30c-4e9f-97b3-844126fcb52e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.218766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.220046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.220739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.221186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.223241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:31:03.223622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.225695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.245185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 905/1827] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/projects => generated 391 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:31:03.246819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.250242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 923/1828] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:31:03.251891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.268272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a299ab9-5ac5-46d0-bb8c-3759cb7fa5b8 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.338442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ae0d7-25c6-4092-a4cd-7caa5b2d27fe admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.344118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.345412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.345709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.345979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.346849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.349034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.361634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.362252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.362533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.362796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.363518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.365729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.384540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:03.384946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 906/1829] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/users => generated 296 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:03.391706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.403163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15ccff-4ef4-42b6-922f-e248336ce1e0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.410020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 924/1830] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/projects => generated 359 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:03.411007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.414323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:03.509929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.511705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.512266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.512574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.513488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.515345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727320b6-8893-4a4c-89bf-fbf4a4cf830e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.524598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.529071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.532916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.533205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.536450 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 907/1831] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:03.536821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.536821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.536821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.551294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.567790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 925/1832] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/users => generated 280 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:31:03.571822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.576031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1f3e85-7411-4e1c-985a-b15fdec7d7e8 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.583236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c95cc37-8618-4628-8ded-d29e0d6296a6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.592978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.593823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.594226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.594499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.595212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.599209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.601631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.602315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.602594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.602862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.603528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.605220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.610909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 926/1833] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:31:03.614952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.617374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 908/1834] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:03.621459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.625247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b90b971-35a7-423a-8d8f-9897dc2f9bd2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.630880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.631582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.631953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.632316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.633088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.633184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365162fc-a21a-4bac-938f-9c3c5f60df4e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.634925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.638024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba060e8-d7b8-439a-9351-609106fe0960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.638791 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba060e8-d7b8-439a-9351-609106fe0960 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.639071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba060e8-d7b8-439a-9351-609106fe0960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.639344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba060e8-d7b8-439a-9351-609106fe0960 admin admin] PATH_INFO: `/v3/projects/c377620bddd4454eae9143c76dd2cf7f/users/43354b7aa0ce4a65a7a511d6a5101cc5/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.646677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 927/1835] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:31:03.651005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.655178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba060e8-d7b8-439a-9351-609106fe0960 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c377620bddd4454eae9143c76dd2cf7f, user_id=43354b7aa0ce4a65a7a511d6a5101cc5, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.658421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba060e8-d7b8-439a-9351-609106fe0960 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.662090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576e82b6-738e-4180-9a28-a4de36a65b8a admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.667400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df25c820-1c03-42d5-b342-3caca1294999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.668052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df25c820-1c03-42d5-b342-3caca1294999 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.668323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df25c820-1c03-42d5-b342-3caca1294999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.668589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df25c820-1c03-42d5-b342-3caca1294999 admin admin] PATH_INFO: `/v3/projects/6fe98316c130442fabcbe2f360725b9b/users/96d0dd2a936c4afd8803322f679237ba/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.672749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 909/1836] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:03 2022] PUT /identity/v3/projects/c377620bddd4454eae9143c76dd2cf7f/users/43354b7aa0ce4a65a7a511d6a5101cc5/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:31:03.684024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.684307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.684575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.686580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df25c820-1c03-42d5-b342-3caca1294999 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fe98316c130442fabcbe2f360725b9b, user_id=96d0dd2a936c4afd8803322f679237ba, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.689394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df25c820-1c03-42d5-b342-3caca1294999 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.702286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 928/1837] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:03 2022] PUT /identity/v3/projects/6fe98316c130442fabcbe2f360725b9b/users/96d0dd2a936c4afd8803322f679237ba/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:03.712128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] MFA Rules not processed for user `43354b7aa0ce4a65a7a511d6a5101cc5`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:03.713375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.713488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.713727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.713994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.742977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] MFA Rules not processed for user `96d0dd2a936c4afd8803322f679237ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:03.744604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.782044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 910/1838] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:03.794616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.796428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.820376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 929/1839] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:03 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:31:03.856062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805cb41a-046c-4db4-8bed-b1a8890c056d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.856227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.857668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:03.859782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.860560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.860866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.861135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.869956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.870218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926c8d7c-3b21-4a21-9c3e-0f1ee80b8a6a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.872232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.881653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6794fa1-fd7e-4570-be00-99b6d537266e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:03.881963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 911/1840] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:31:03.882681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6794fa1-fd7e-4570-be00-99b6d537266e service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:03.882907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6794fa1-fd7e-4570-be00-99b6d537266e service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:03.883277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6794fa1-fd7e-4570-be00-99b6d537266e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:03.892672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6794fa1-fd7e-4570-be00-99b6d537266e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:03.894127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6794fa1-fd7e-4570-be00-99b6d537266e service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:03.903298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 930/1841] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:03 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:31:03.920435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:03.929497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:03.994407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8a69c1-8563-4563-8fed-8d81374cb8bc service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:03.999313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:04.000046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:04.000540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:04.000936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] PATH_INFO: `/v3/users/96d0dd2a936c4afd8803322f679237ba` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:04.002744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96d0dd2a936c4afd8803322f679237ba)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:04.004916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:04.065838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96d0dd2a936c4afd8803322f679237ba'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.081885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96d0dd2a936c4afd8803322f679237ba'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.082532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96d0dd2a936c4afd8803322f679237ba'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.092543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96d0dd2a936c4afd8803322f679237ba'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.093448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96d0dd2a936c4afd8803322f679237ba'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.125048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 912/1842] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:03 2022] DELETE /identity/v3/users/96d0dd2a936c4afd8803322f679237ba => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:04.126786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:04.127071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:04.127338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:04.148769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:04.150963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:04.160884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:04.241356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 931/1843] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:04.251783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:04.258267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:04.322860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de7e52-b3dd-434f-86af-4286d2638398 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:04.326732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:04.327338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:04.327608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:04.327872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:04.337985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:04.339448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:04.349833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 913/1844] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:04.617320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:04.620034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:04.637828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1a9f7a-0875-42dc-b144-3ee8a88dc876 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:04.644312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:04.644981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:04.645264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:04.646228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] PATH_INFO: `/v3/projects/6fe98316c130442fabcbe2f360725b9b` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:04.647366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6fe98316c130442fabcbe2f360725b9b)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:04.649464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:04.723610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6fe98316c130442fabcbe2f360725b9b', 'request_id': 'req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.729567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6fe98316c130442fabcbe2f360725b9b', 'request_id': 'req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.739809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6fe98316c130442fabcbe2f360725b9b', 'request_id': 'req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.765582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] The token cache is being invalidate because project 6fe98316c130442fabcbe2f360725b9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:04.765940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.768092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-40aebc0b-47bb-41c9-aea6-0cc76ccb2268 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:04.770929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 932/1845] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:04 2022] DELETE /identity/v3/projects/6fe98316c130442fabcbe2f360725b9b => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:31:09.019276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:09.020673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:09.053643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:09.081596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e79ba3e-8515-4c1e-924a-e61ca87f5715 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:09.132370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:09.133015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:09.133291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:09.133559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:09.135279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:09.198675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:09.200117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:09.210133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 914/1846] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:09 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:14.759337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:19.208159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=57 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:41.173388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:41.173807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:41.174010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:41.205386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:41.207078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:41.307866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 933/1847] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:31:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:41.321670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:41.323242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:41.392223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:41.394210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:41.432740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9b6109-4349-405f-b9c3-1aa7f9763d8a service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:41.432740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea2935-f118-4c4f-9b38-a6d3493c1a70 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:41.432740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea2935-f118-4c4f-9b38-a6d3493c1a70 service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:41.432740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea2935-f118-4c4f-9b38-a6d3493c1a70 service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:41.441355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea2935-f118-4c4f-9b38-a6d3493c1a70 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:41.467892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea2935-f118-4c4f-9b38-a6d3493c1a70 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:41.471328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea2935-f118-4c4f-9b38-a6d3493c1a70 service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:41.488594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 915/1848] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:31:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:31:41.512659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97498417-68d7-48f4-911e-69bb3e93f7da None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:41.518133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:41.518781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:41.519056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:41.519326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] PATH_INFO: `/v3/users/43354b7aa0ce4a65a7a511d6a5101cc5` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:41.520482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43354b7aa0ce4a65a7a511d6a5101cc5)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:41.522493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:41.637813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43354b7aa0ce4a65a7a511d6a5101cc5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:41.654503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43354b7aa0ce4a65a7a511d6a5101cc5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:41.655419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43354b7aa0ce4a65a7a511d6a5101cc5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:41.665916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43354b7aa0ce4a65a7a511d6a5101cc5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:41.667514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43354b7aa0ce4a65a7a511d6a5101cc5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:41.686208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 934/1849] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:41 2022] DELETE /identity/v3/users/43354b7aa0ce4a65a7a511d6a5101cc5 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:31:41.691249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:41.691535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:41.691807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:41.716764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:41.718215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:41.799844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 916/1850] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:41.811346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:41.814024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:41.851107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:41.886781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb00e89-6b28-4551-828f-0e2c82b486e8 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:41.890802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:41.891581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:41.891768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:41.892009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:41.901754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:41.903177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:41.914312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 935/1851] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:31:42.249448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:42.250899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:42.276920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd6119-2c50-43c0-9c4b-a12d6ce0170f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:42.285041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:42.286048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:42.292841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:42.293338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] PATH_INFO: `/v3/projects/c377620bddd4454eae9143c76dd2cf7f` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:42.296398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c377620bddd4454eae9143c76dd2cf7f)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:42.299791 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:42.370045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c377620bddd4454eae9143c76dd2cf7f', 'request_id': 'req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:42.377861 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c377620bddd4454eae9143c76dd2cf7f', 'request_id': 'req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:42.385421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c377620bddd4454eae9143c76dd2cf7f', 'request_id': 'req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:42.404102 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] The token cache is being invalidate because project c377620bddd4454eae9143c76dd2cf7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:42.404392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:42.405111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-bfe02e6d-38f9-4d8e-a18d-789bd21853e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:42.409114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 917/1852] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:42 2022] DELETE /identity/v3/projects/c377620bddd4454eae9143c76dd2cf7f => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:50.331510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:50.349718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:50.365755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0621da48-30b9-4d25-b305-1407f01496fb service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:50.459998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:50.460575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:50.460885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:50.461162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] PATH_INFO: `/v3/users/c7bbcf11897b45229f3eebb7d8f2e2b0` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:50.462376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7bbcf11897b45229f3eebb7d8f2e2b0)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:50.464422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:50.522889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7bbcf11897b45229f3eebb7d8f2e2b0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:50.545161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7bbcf11897b45229f3eebb7d8f2e2b0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:50.546441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7bbcf11897b45229f3eebb7d8f2e2b0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:50.584352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7bbcf11897b45229f3eebb7d8f2e2b0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:50.585232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7bbcf11897b45229f3eebb7d8f2e2b0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:50.603389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 936/1853] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:50 2022] DELETE /identity/v3/users/c7bbcf11897b45229f3eebb7d8f2e2b0 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:50.614261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:50.614896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:50.614896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:50.635200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:50.636447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:50.652076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:50.722131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 918/1854] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:50.733482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:50.735048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:50.806391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:50.819186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34c904b-764c-4426-a33b-f05fe7572146 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:50.829474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:50.830100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:50.830372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:50.830660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:50.840822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:50.842502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:50.853570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 937/1855] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:51.395765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:51.397417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:51.415278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f13c78-f463-4ec3-8e9e-e06020f3e08f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:51.423233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:51.424155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:51.424422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:51.424835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] PATH_INFO: `/v3/projects/d1067f8f40df4805a9b9dc16a2dc7eb1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:51.426640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1067f8f40df4805a9b9dc16a2dc7eb1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:51.428891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:51.488613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1067f8f40df4805a9b9dc16a2dc7eb1', 'request_id': 'req-766cad5e-5f20-48eb-bd2f-db644b432a01'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.496145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1067f8f40df4805a9b9dc16a2dc7eb1', 'request_id': 'req-766cad5e-5f20-48eb-bd2f-db644b432a01'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.504371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1067f8f40df4805a9b9dc16a2dc7eb1', 'request_id': 'req-766cad5e-5f20-48eb-bd2f-db644b432a01'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.520395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] The token cache is being invalidate because project d1067f8f40df4805a9b9dc16a2dc7eb1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:51.520583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.521268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.523014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 919/1856] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:51 2022] DELETE /identity/v3/projects/d1067f8f40df4805a9b9dc16a2dc7eb1 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:51.527043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:51.537520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05a9616-dd5b-4aba-9f7e-faae1f72bc8c service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:51.600452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:51.601229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:51.601504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:51.601774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] PATH_INFO: `/v3/users/7264a0f671b344fbaee53b69d29e7613` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:51.602898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7264a0f671b344fbaee53b69d29e7613)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:51.604855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:51.654009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7264a0f671b344fbaee53b69d29e7613'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.669587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7264a0f671b344fbaee53b69d29e7613'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.670693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7264a0f671b344fbaee53b69d29e7613'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.680813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7264a0f671b344fbaee53b69d29e7613'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.681452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0ae02369-b9cd-479b-9fe4-1ae2cd121d3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7264a0f671b344fbaee53b69d29e7613'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:51.695787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 938/1857] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:51 2022] DELETE /identity/v3/users/7264a0f671b344fbaee53b69d29e7613 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:52.013677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.015712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:52.041437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766cad5e-5f20-48eb-bd2f-db644b432a01 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.109231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.109877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.110219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.110481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] PATH_INFO: `/v3/projects/422cc110086d4572a85f957f1d02505a` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.111645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=422cc110086d4572a85f957f1d02505a)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.114141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.165233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:52.174737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '422cc110086d4572a85f957f1d02505a', 'request_id': 'req-85bd6014-f885-493f-a808-8cb72b7c1d4f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:52.183106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '422cc110086d4572a85f957f1d02505a', 'request_id': 'req-85bd6014-f885-493f-a808-8cb72b7c1d4f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:52.198594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '422cc110086d4572a85f957f1d02505a', 'request_id': 'req-85bd6014-f885-493f-a808-8cb72b7c1d4f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:52.219264 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] The token cache is being invalidate because project 422cc110086d4572a85f957f1d02505a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:52.219558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:52.220231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-85bd6014-f885-493f-a808-8cb72b7c1d4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:52.226481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 920/1858] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:52 2022] DELETE /identity/v3/projects/422cc110086d4572a85f957f1d02505a => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:52.231714 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.231999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.232278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.257137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:52.258462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:52.330537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 939/1859] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:52.339804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.340456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.340456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.373306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:52.377263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:52.386679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 921/1860] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:52.391009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.404274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f6514b-2632-4b6c-8103-6cf1803806a7 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.410953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.411726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.412106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.412568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.413435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:31:52.414273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.417235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.420296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:52.438629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.438773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 940/1861] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:31:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:31:52.450062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11a7f81-ceaf-4303-bc75-efd9c76a1c5e None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.464075 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.464733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.465001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.465417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.465906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.467895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.529857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.537947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 922/1862] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:31:52 2022] POST /identity/v3/projects => generated 381 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:31:52.541813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be111c1-a323-4484-8741-4ca1fa738563 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.615392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.616111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.616587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.616587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.618105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.619267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.663470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 941/1863] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:31:52 2022] POST /identity/v3/users => generated 291 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:52.669653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.681707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915847ff-591d-4f64-98d6-0ef9674c632d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.687628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.688363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.688740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.689156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.689944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.691781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.704245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 923/1864] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:52 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:52.710064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.723755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a1f38-480e-42ad-9740-bcc22be74426 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.729641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70244e84-6887-4383-ad77-5ffe69eccf2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.729641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70244e84-6887-4383-ad77-5ffe69eccf2b admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.729893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70244e84-6887-4383-ad77-5ffe69eccf2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.730115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70244e84-6887-4383-ad77-5ffe69eccf2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.731357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70244e84-6887-4383-ad77-5ffe69eccf2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.733943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70244e84-6887-4383-ad77-5ffe69eccf2b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.748064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 942/1865] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:52 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:52.752301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:52.767641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee0790-bdbf-44d8-b736-9055dd54ac27 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:52.773491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:52.774257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.774606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.774951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] PATH_INFO: `/v3/projects/cce52b162ed34452b86fc4d193463a98/users/6f193164f20d4cee949eca83dc0d1f00/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.792589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cce52b162ed34452b86fc4d193463a98, user_id=6f193164f20d4cee949eca83dc0d1f00, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:52.795314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:52.811359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 924/1866] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:52 2022] PUT /identity/v3/projects/cce52b162ed34452b86fc4d193463a98/users/6f193164f20d4cee949eca83dc0d1f00/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:52.826051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:52.826549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:52.827566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:52.861365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] MFA Rules not processed for user `6f193164f20d4cee949eca83dc0d1f00`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:52.863216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:52.942934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 943/1867] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:52.947518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.016511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ea832-0261-4fc9-83bc-7fa11b63b18b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.023624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.024413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.024649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.024805 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.025447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.027386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.058399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 925/1868] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:31:52 2022] POST /identity/v3/projects => generated 381 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:53.063093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.075114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7f794-fed5-447f-92aa-63a13737db79 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.153437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.154294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.155178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.155601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.156250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.159109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.170049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:53.184518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 944/1869] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:31:53 2022] POST /identity/v3/users => generated 297 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:31:53.188878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.200139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3719de-befc-463e-be5c-28bceb5d7f3c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.205232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.206190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.206468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.206744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.207392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.209148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.225232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 926/1870] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:53 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:31:53.226892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.238332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b45408-1e84-4d4d-9989-6c94470c4323 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.243214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.243884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.244169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.244426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] PATH_INFO: `/v3/projects/e8038c6d565a42049646132853d87205/users/e4a44d1007fa43f48920bc4009315ad1/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.260266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8038c6d565a42049646132853d87205, user_id=e4a44d1007fa43f48920bc4009315ad1, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.263169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.279421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 945/1871] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:53 2022] PUT /identity/v3/projects/e8038c6d565a42049646132853d87205/users/e4a44d1007fa43f48920bc4009315ad1/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:53.283841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.305320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435a7904-eb80-483b-9e2e-74a5c322cb25 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.340642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:53.373223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e82716-4208-4576-a24e-9e76a79c408a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.374124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e82716-4208-4576-a24e-9e76a79c408a admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.374613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e82716-4208-4576-a24e-9e76a79c408a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.374892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e82716-4208-4576-a24e-9e76a79c408a admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.375638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e82716-4208-4576-a24e-9e76a79c408a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.377575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e82716-4208-4576-a24e-9e76a79c408a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.389611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 927/1872] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:31:53 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:31:53.397497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.419940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c7b76d-e082-4c45-ad6a-6123e5009915 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.425495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.426221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.426687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.426889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] PATH_INFO: `/v3/projects/e8038c6d565a42049646132853d87205/users/e4a44d1007fa43f48920bc4009315ad1/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.429031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8038c6d565a42049646132853d87205, user_id=e4a44d1007fa43f48920bc4009315ad1, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.431757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.448781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 946/1873] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:31:53 2022] PUT /identity/v3/projects/e8038c6d565a42049646132853d87205/users/e4a44d1007fa43f48920bc4009315ad1/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:53.456455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.456655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.458635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.491560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] MFA Rules not processed for user `e4a44d1007fa43f48920bc4009315ad1`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:53.492844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:53.618443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 928/1874] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:53 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:53.655819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.664076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:53.757296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076f8b4d-9d6d-4754-b35b-b370a54d24a9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.762988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1756701-ce8c-4c5e-ae5d-dd76cbf44ab7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.763879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1756701-ce8c-4c5e-ae5d-dd76cbf44ab7 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.764344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1756701-ce8c-4c5e-ae5d-dd76cbf44ab7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.764806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1756701-ce8c-4c5e-ae5d-dd76cbf44ab7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:53.780863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1756701-ce8c-4c5e-ae5d-dd76cbf44ab7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:53.782328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1756701-ce8c-4c5e-ae5d-dd76cbf44ab7 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:53.808893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 947/1875] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:53 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:53.948407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:53.958702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6699b66-2276-41f2-a6fc-05460c490a72 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:53.962571 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:53.963277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:53.963500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:53.963821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:54.040017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:54.041813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:54.054602 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 929/1876] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:53 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:31:55.262840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:55.263391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:55.263391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:55.288606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:55.289838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:55.357730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 948/1877] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:31:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:31:55.377139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:55.379088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:55.454981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:55.475073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082a0fba-f4ad-4baa-9eb5-97a4c0f51caf service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:55.480875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3a5270-e199-4897-936f-3e99e9d548f1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:55.481595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3a5270-e199-4897-936f-3e99e9d548f1 service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:55.482297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3a5270-e199-4897-936f-3e99e9d548f1 service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:55.482865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3a5270-e199-4897-936f-3e99e9d548f1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:55.499790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3a5270-e199-4897-936f-3e99e9d548f1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:55.501895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3a5270-e199-4897-936f-3e99e9d548f1 service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:55.513447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 930/1878] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:31:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:31:57.049138 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:57.194946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cae123e-63d6-49ad-b073-1942488e04e8 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:57.215397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:57.216285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:57.217083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:57.218293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] PATH_INFO: `/v3/users/6f193164f20d4cee949eca83dc0d1f00` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:57.222357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f193164f20d4cee949eca83dc0d1f00)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:57.226713 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:57.232964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:57.322841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f193164f20d4cee949eca83dc0d1f00'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:57.380756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f193164f20d4cee949eca83dc0d1f00'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:57.385310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f193164f20d4cee949eca83dc0d1f00'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:57.411449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f193164f20d4cee949eca83dc0d1f00'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:57.412425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f193164f20d4cee949eca83dc0d1f00'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:57.468582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 949/1879] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:57 2022] DELETE /identity/v3/users/6f193164f20d4cee949eca83dc0d1f00 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:57.469171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:57.469914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:57.469914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:57.511026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:57.513736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:57.698683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 931/1880] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:31:57.714826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:57.716462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:57.889717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d880ca68-0194-4843-a46a-1e66f8968001 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:57.896882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:57.899971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:57.900563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:57.900962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:57.914397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:57.916014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:57.927609 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 950/1881] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:31:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:31:58.462770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:58.464355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:58.483890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a09e0f-f4b3-4dff-8fd4-60097e665fdf None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:58.518118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:58.519604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:58.520051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:58.520471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] PATH_INFO: `/v3/projects/cce52b162ed34452b86fc4d193463a98` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:58.522403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cce52b162ed34452b86fc4d193463a98)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:58.525705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:58.619827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cce52b162ed34452b86fc4d193463a98', 'request_id': 'req-30fd923a-99bb-4bd4-90af-262ec9a45372'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:58.630341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cce52b162ed34452b86fc4d193463a98', 'request_id': 'req-30fd923a-99bb-4bd4-90af-262ec9a45372'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:58.644809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cce52b162ed34452b86fc4d193463a98', 'request_id': 'req-30fd923a-99bb-4bd4-90af-262ec9a45372'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:58.672267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] The token cache is being invalidate because project cce52b162ed34452b86fc4d193463a98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:58.672874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:58.674992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:58.681533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 932/1882] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:58 2022] DELETE /identity/v3/projects/cce52b162ed34452b86fc4d193463a98 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:58.685288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:58.708144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0af4a8-9c9f-4c89-9dd9-467fb0ec48be service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:58.922804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:58.923469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:58.926032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:58.926360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] PATH_INFO: `/v3/users/e4a44d1007fa43f48920bc4009315ad1` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:58.930940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4a44d1007fa43f48920bc4009315ad1)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:58.932945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:59.019516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:59.061528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4a44d1007fa43f48920bc4009315ad1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.084642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4a44d1007fa43f48920bc4009315ad1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.085775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4a44d1007fa43f48920bc4009315ad1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.105189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4a44d1007fa43f48920bc4009315ad1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.106048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-7469df04-8cc1-4387-9387-549be87c98a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4a44d1007fa43f48920bc4009315ad1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.136673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 951/1883] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:31:58 2022] DELETE /identity/v3/users/e4a44d1007fa43f48920bc4009315ad1 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:59.620043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:31:59.621910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:31:59.688167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fd923a-99bb-4bd4-90af-262ec9a45372 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:31:59.770659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:31:59.771466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:59.771765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:59.772011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] PATH_INFO: `/v3/projects/e8038c6d565a42049646132853d87205` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:59.773241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8038c6d565a42049646132853d87205)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:31:59.775667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:31:59.806034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:31:59.861285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8038c6d565a42049646132853d87205', 'request_id': 'req-9812a993-36ce-459c-a0e4-7729934bf932'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.889137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8038c6d565a42049646132853d87205', 'request_id': 'req-9812a993-36ce-459c-a0e4-7729934bf932'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.896978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8038c6d565a42049646132853d87205', 'request_id': 'req-9812a993-36ce-459c-a0e4-7729934bf932'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.916544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] The token cache is being invalidate because project e8038c6d565a42049646132853d87205 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:31:59.917108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.917970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9812a993-36ce-459c-a0e4-7729934bf932 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:31:59.919891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 933/1884] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:31:59 2022] DELETE /identity/v3/projects/e8038c6d565a42049646132853d87205 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:31:59.927008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:31:59.927890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:31:59.929156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:31:59.980346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:31:59.980639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:00.092679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 952/1885] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:31:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:00.106074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.106355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.106626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.155612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:00.156859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:00.168327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 934/1886] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:00.169859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.181641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eefab-bfb5-4b47-9529-265348cfab26 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.186990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.187665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.187986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.188335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.189045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:32:00.189518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.191268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.206824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 953/1887] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:32:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:32:00.211903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.223027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3311323e-1924-404b-81cb-b4aff0660624 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.227428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.228189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.228480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.228776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.229993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.231926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.290528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 935/1888] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/projects => generated 387 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:00.294296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.311435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bf484-a6bd-4637-a8b3-57cd676290fb admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.406383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.407014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.407291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.407581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.408222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.410690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.459483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 954/1889] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/users => generated 294 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:00.463322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.475672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561cd374-4778-44da-9420-a7db369fdceb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.480674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.481382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.481666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.481936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.483293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.484989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.496851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 936/1890] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:00.501535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.512223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f32f955-35a3-46ba-8ad7-2b1b1cc0641d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.517667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f616e3b-785d-4532-8650-cdaef3a2b7f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.518330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f616e3b-785d-4532-8650-cdaef3a2b7f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.518614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f616e3b-785d-4532-8650-cdaef3a2b7f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.518883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f616e3b-785d-4532-8650-cdaef3a2b7f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.519524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f616e3b-785d-4532-8650-cdaef3a2b7f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.521240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f616e3b-785d-4532-8650-cdaef3a2b7f0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.534486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 955/1891] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:32:00.538570 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.549742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf3cafc-40f8-41fc-9bc7-941d277e0bf0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.554913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.555670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.556012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.556353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] PATH_INFO: `/v3/projects/aaa5b37c77ca4474bf07106832af4b66/users/59e7a81f432f4c69aacb3b01943dab3d/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.575319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaa5b37c77ca4474bf07106832af4b66, user_id=59e7a81f432f4c69aacb3b01943dab3d, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.578096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.596889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 937/1892] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:00 2022] PUT /identity/v3/projects/aaa5b37c77ca4474bf07106832af4b66/users/59e7a81f432f4c69aacb3b01943dab3d/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:00.605524 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f5c77e-07ff-43c9-bce1-ae484283ff11 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.607190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f5c77e-07ff-43c9-bce1-ae484283ff11 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.607190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f5c77e-07ff-43c9-bce1-ae484283ff11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.629155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:00.638468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-15f5c77e-07ff-43c9-bce1-ae484283ff11 None None] MFA Rules not processed for user `59e7a81f432f4c69aacb3b01943dab3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:00.641036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-15f5c77e-07ff-43c9-bce1-ae484283ff11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:00.722680 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 956/1893] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:00.723614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.806524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6009f-6bde-420b-8130-e4d63c0bbae7 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.812344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.812992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.813267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.813559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.814476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.817194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:00.867238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 938/1894] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/projects => generated 387 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:00.872028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.872982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.873790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.874055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:00.913212 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:00.927003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:00.928245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:00.972451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f49636-7f83-45b7-92b6-4e8005e4d6ae admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:00.981154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:00.983074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:00.983469 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:00.983817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:00.984526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:00.987361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.020764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 939/1895] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/users => generated 300 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:01.023510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:01.037179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feff964-6bb8-4dca-a261-e6946867c051 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:01.039822 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 957/1896] 10.222.0.23 () {58 vars in 967 bytes} [Wed Oct 12 02:32:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:01.044082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:01.045525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.048654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.049073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.049817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:01.051648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.057210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:01.058641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:01.065258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 940/1897] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:01.069424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:01.084269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e685e265-1e60-498d-9126-bdc6b710a832 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:01.090454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:01.091207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.091556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.092508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] PATH_INFO: `/v3/projects/d0042265978f4c42a70c681d019cffb5/users/a50f030e3a6140ceb1f1fe748c193e4c/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.117467 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0042265978f4c42a70c681d019cffb5, user_id=a50f030e3a6140ceb1f1fe748c193e4c, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:01.122980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.144230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 941/1898] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:01 2022] PUT /identity/v3/projects/d0042265978f4c42a70c681d019cffb5/users/a50f030e3a6140ceb1f1fe748c193e4c/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:01.148035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:01.186849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a6d70f-c947-4fcc-8f68-f29f2c0a7e9e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:01.248410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:01.283620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:01.284262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.284539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.285112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.294435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e059a0c0-b937-4450-b058-a336991b51b6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:01.305977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:01.306611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.306891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.307161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.307812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:01.310447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.324059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 942/1899] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:01.329547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:01.341101 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b2041-631c-4432-b8a2-e635c53be167 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:01.346748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d384d66-a21a-483f-8419-981e0ea7cc96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:01.347418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d384d66-a21a-483f-8419-981e0ea7cc96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.347699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d384d66-a21a-483f-8419-981e0ea7cc96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.347973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d384d66-a21a-483f-8419-981e0ea7cc96 admin admin] PATH_INFO: `/v3/projects/d0042265978f4c42a70c681d019cffb5/users/a50f030e3a6140ceb1f1fe748c193e4c/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.350239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d384d66-a21a-483f-8419-981e0ea7cc96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0042265978f4c42a70c681d019cffb5, user_id=a50f030e3a6140ceb1f1fe748c193e4c, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:01.353138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d384d66-a21a-483f-8419-981e0ea7cc96 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.377395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 943/1900] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:01 2022] PUT /identity/v3/projects/d0042265978f4c42a70c681d019cffb5/users/a50f030e3a6140ceb1f1fe748c193e4c/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:01.381876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.382156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.383483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.406415 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:01.408026 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.428033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 958/1901] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:32:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:32:01.436328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] MFA Rules not processed for user `a50f030e3a6140ceb1f1fe748c193e4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:01.437587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:01.530862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:01.569262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 944/1902] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:01 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:01.580605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:01.582329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:01.673740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73015b6c-26f9-4133-be86-c2b491efa366 service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:01.746182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0c066-40d5-48d6-8972-175c5ab53a18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:01.746887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0c066-40d5-48d6-8972-175c5ab53a18 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:01.747167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0c066-40d5-48d6-8972-175c5ab53a18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:01.747516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0c066-40d5-48d6-8972-175c5ab53a18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:01.760645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0c066-40d5-48d6-8972-175c5ab53a18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:01.762194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0c066-40d5-48d6-8972-175c5ab53a18 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:01.773222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 959/1903] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:01 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:01.958610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:02.087027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cc7295-b311-4014-b166-9d1d90338bfd None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:02.094025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:02.094849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:02.095371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:02.095952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] PATH_INFO: `/v3/users/59e7a81f432f4c69aacb3b01943dab3d` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:02.106270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] RBAC: Authorizing `identity:delete_user(user_id=59e7a81f432f4c69aacb3b01943dab3d)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:02.108683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:02.168129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59e7a81f432f4c69aacb3b01943dab3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:02.182468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59e7a81f432f4c69aacb3b01943dab3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:02.183331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59e7a81f432f4c69aacb3b01943dab3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:02.204259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59e7a81f432f4c69aacb3b01943dab3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:02.205320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59e7a81f432f4c69aacb3b01943dab3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:02.226707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 945/1904] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:01 2022] DELETE /identity/v3/users/59e7a81f432f4c69aacb3b01943dab3d => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:02.228985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:02.229442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:02.229442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:02.266406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:02.267967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:02.318490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:02.343178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 960/1905] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:02.349510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:02.351239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:02.441038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eba99ad-7766-4b54-8e71-fb7f9efed78f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:02.446608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:02.447601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:02.447876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:02.448143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:02.458790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:02.460248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:02.471506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 946/1906] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:03.318073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:03.319643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:03.351968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211e473-cc3d-4582-92f3-5b59557397b0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:03.365154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:03.370367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:03.371492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:03.371992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] PATH_INFO: `/v3/projects/aaa5b37c77ca4474bf07106832af4b66` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:03.373965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaa5b37c77ca4474bf07106832af4b66)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:03.376994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:03.452104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaa5b37c77ca4474bf07106832af4b66', 'request_id': 'req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.472492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaa5b37c77ca4474bf07106832af4b66', 'request_id': 'req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.484847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaa5b37c77ca4474bf07106832af4b66', 'request_id': 'req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.498821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] The token cache is being invalidate because project aaa5b37c77ca4474bf07106832af4b66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:03.499216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.499974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.501840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 961/1907] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:03 2022] DELETE /identity/v3/projects/aaa5b37c77ca4474bf07106832af4b66 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:03.505598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:03.516730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce4f08c-4584-4c0e-aa07-fe2084206773 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:03.520858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:03.586924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:03.587731 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:03.588221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:03.588466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] PATH_INFO: `/v3/users/a50f030e3a6140ceb1f1fe748c193e4c` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:03.590215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a50f030e3a6140ceb1f1fe748c193e4c)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:03.592925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:03.661700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a50f030e3a6140ceb1f1fe748c193e4c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.678623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a50f030e3a6140ceb1f1fe748c193e4c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.679794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a50f030e3a6140ceb1f1fe748c193e4c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.689646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a50f030e3a6140ceb1f1fe748c193e4c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.692918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a50f030e3a6140ceb1f1fe748c193e4c'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:03.710994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 947/1908] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:03 2022] DELETE /identity/v3/users/a50f030e3a6140ceb1f1fe748c193e4c => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:03.969310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:03.971536 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.009351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d1f3d-2d68-4d8d-9810-a4c5d9786f80 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:04.133683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:04.135898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.145959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:04.147346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.147832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.148332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] PATH_INFO: `/v3/users/a2ce151a58444f84b06a6e10cdebf1c3` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.149745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2ce151a58444f84b06a6e10cdebf1c3)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:04.151925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:04.216317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888f723b-010c-4aa5-ac15-7d4097745ce4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:04.238172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2ce151a58444f84b06a6e10cdebf1c3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.238414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:04.239084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.239362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.239632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] PATH_INFO: `/v3/projects/d0042265978f4c42a70c681d019cffb5` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.241100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0042265978f4c42a70c681d019cffb5)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:04.243182 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:04.300740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2ce151a58444f84b06a6e10cdebf1c3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.300740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2ce151a58444f84b06a6e10cdebf1c3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.305676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2ce151a58444f84b06a6e10cdebf1c3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.306429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a9d9bfd8-94af-445a-a47f-5f514f5e2d39 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2ce151a58444f84b06a6e10cdebf1c3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.310492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:04.341668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 962/1909] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:03 2022] DELETE /identity/v3/users/a2ce151a58444f84b06a6e10cdebf1c3 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:04.366610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0042265978f4c42a70c681d019cffb5', 'request_id': 'req-fc8ae559-6b2b-453a-b897-fdcff81d657a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.379322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0042265978f4c42a70c681d019cffb5', 'request_id': 'req-fc8ae559-6b2b-453a-b897-fdcff81d657a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.390777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0042265978f4c42a70c681d019cffb5', 'request_id': 'req-fc8ae559-6b2b-453a-b897-fdcff81d657a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.421371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] The token cache is being invalidate because project d0042265978f4c42a70c681d019cffb5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:04.421721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.422526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-fc8ae559-6b2b-453a-b897-fdcff81d657a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.429407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 948/1910] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:04 2022] DELETE /identity/v3/projects/d0042265978f4c42a70c681d019cffb5 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:04.431269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.431559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.431834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.511159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:04.512776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.668527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 963/1911] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:04.677460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.678435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.678907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.692041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:04.693489 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.715167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4272ff60-0dee-42a2-871f-1e9ea531b1a5 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:04.716034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:04.717615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.721917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:04.722566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.723108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.723404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] PATH_INFO: `/v3/projects/177af31ad03c40e6b2a4a72c16451b5d` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.727463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=177af31ad03c40e6b2a4a72c16451b5d)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:04.730811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:04.741882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 949/1912] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:04.743312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:04.751552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:04.764738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1360eb-863c-4a6f-a1d0-2a65eabeca96 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:04.781481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:04.782276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.782461 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.782739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.783690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:32:04.784108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:04.785880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:04.811655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 950/1913] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:32:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:32:04.816649 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:04.830013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2f9998-8544-48c4-acd7-4a30291d1bc5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:04.833533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '177af31ad03c40e6b2a4a72c16451b5d', 'request_id': 'req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.841244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '177af31ad03c40e6b2a4a72c16451b5d', 'request_id': 'req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.852869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '177af31ad03c40e6b2a4a72c16451b5d', 'request_id': 'req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.867461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] The token cache is being invalidate because project 177af31ad03c40e6b2a4a72c16451b5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:04.867765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.868433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-12332a25-4a40-4c92-ae0f-4cbe06e7ad85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:04.873968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 964/1914] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:04 2022] DELETE /identity/v3/projects/177af31ad03c40e6b2a4a72c16451b5d => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:04.875129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21dd3f8-1bac-4a8a-a292-c111008619e9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.875420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21dd3f8-1bac-4a8a-a292-c111008619e9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.875697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21dd3f8-1bac-4a8a-a292-c111008619e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.904813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:04.905768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.906432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.906941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.907954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:04.910711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:04.911426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e21dd3f8-1bac-4a8a-a292-c111008619e9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:04.915050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e21dd3f8-1bac-4a8a-a292-c111008619e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.934065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 965/1915] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:04.936094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:04.936451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:04.936782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:04.955819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 951/1916] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:04 2022] POST /identity/v3/projects => generated 379 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:04.971933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:04.974841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:04.978891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:04.980084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:05.079002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096662d7-bdd3-4ac4-a189-281f3b1800cc admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.093911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.094924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.095443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.095930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.097138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.099426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.130087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 966/1917] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:05.135104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 952/1918] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:04 2022] POST /identity/v3/users => generated 290 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:05.135510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.136317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.146056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f56410-6cad-4384-9ffb-c993e7af7702 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.148204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d2f7e-a4c8-4f8d-9d1f-4d6231bdb962 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.151086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.151701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.151973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.152246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.152915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:32:05.153455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.155373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.155632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.156194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.156567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.156970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.157728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.158673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:05.160545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.171208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 967/1919] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:32:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:32:05.176092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.177763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 953/1920] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:05.181645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.191836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950a53d6-f486-4858-a977-6a0e3c619c30 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.193952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc270002-7e0c-4470-87c7-556cb229048f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.197152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.197856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.198139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.198405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.199219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.200268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.201148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.201269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.201520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.201858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.202615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.204308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.220763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 954/1921] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:32:05.225558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.245915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91aa29da-022f-4187-8c62-23bf042d07cb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.251477 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0101a172-5158-4478-af76-772c2ce4e76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.252041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 968/1922] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:05 2022] POST /identity/v3/projects => generated 385 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:05.252249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0101a172-5158-4478-af76-772c2ce4e76c admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.252522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0101a172-5158-4478-af76-772c2ce4e76c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.252795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0101a172-5158-4478-af76-772c2ce4e76c admin admin] PATH_INFO: `/v3/projects/052ec677934c4fe4a422d3c857f8ad44/users/6567a4ca408845c4b25593071eea257d/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.256086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.279274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0101a172-5158-4478-af76-772c2ce4e76c admin admin] RBAC: Authorizing `identity:create_grant(project_id=052ec677934c4fe4a422d3c857f8ad44, user_id=6567a4ca408845c4b25593071eea257d, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.282303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0101a172-5158-4478-af76-772c2ce4e76c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.298851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 955/1923] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:05 2022] PUT /identity/v3/projects/052ec677934c4fe4a422d3c857f8ad44/users/6567a4ca408845c4b25593071eea257d/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:05.312876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.313161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.313427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.344879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d3ca58-fdd1-4168-93aa-39cbdbbce766 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.350078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] MFA Rules not processed for user `6567a4ca408845c4b25593071eea257d`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:05.351313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:05.357659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:05.430277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.430890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.431172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.431441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.432078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.434030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.455079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 956/1924] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:05.459988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.463198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 969/1925] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:05 2022] POST /identity/v3/users => generated 293 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:05.468770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.477064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ea039c-b310-457b-bae7-80728c6af82b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.481963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700247f8-b8ce-4ab8-8e81-3fdaa3f8105d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.489381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.491100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.491316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.492048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.492173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.492331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.492620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.492988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.493309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.494324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.494947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.510809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.524721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 970/1926] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:05.528840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.543610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90553d1-444f-4442-a63c-cebc1e1798bc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.564774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.564774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.564774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.564774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.565230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.565230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.582147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 971/1927] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:32:05.583904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.610546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9f4378-c314-4502-a270-a852274958b3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.611499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 957/1928] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:05 2022] POST /identity/v3/projects => generated 379 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:05.617241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.712409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:05.734017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1186e95-99d4-4144-ac11-aa5ab1b33365 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.749702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9e9bf4-80ba-4618-9bdf-85912df52816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.750389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9e9bf4-80ba-4618-9bdf-85912df52816 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.750660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9e9bf4-80ba-4618-9bdf-85912df52816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.750929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9e9bf4-80ba-4618-9bdf-85912df52816 admin admin] PATH_INFO: `/v3/projects/88d84ab9c0dc460085cec44e656f53cd/users/2bff8ece33e9468593d74623841a8497/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.760619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.764685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.765240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.765851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.767223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.770259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.781066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9e9bf4-80ba-4618-9bdf-85912df52816 admin admin] RBAC: Authorizing `identity:create_grant(project_id=88d84ab9c0dc460085cec44e656f53cd, user_id=2bff8ece33e9468593d74623841a8497, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.789702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9e9bf4-80ba-4618-9bdf-85912df52816 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.816311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 958/1929] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:05 2022] POST /identity/v3/users => generated 296 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:05.816460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 972/1930] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:05 2022] PUT /identity/v3/projects/88d84ab9c0dc460085cec44e656f53cd/users/2bff8ece33e9468593d74623841a8497/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:05.820218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:05.828095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935df92b-b601-4a24-9472-3253579e2abd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.828444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935df92b-b601-4a24-9472-3253579e2abd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.828740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935df92b-b601-4a24-9472-3253579e2abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.851271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b38bb5f-18d6-449c-8da4-38c8978ceb35 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:05.873540 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-935df92b-b601-4a24-9472-3253579e2abd None None] MFA Rules not processed for user `2bff8ece33e9468593d74623841a8497`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:05.875115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-935df92b-b601-4a24-9472-3253579e2abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:05.934415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:05.935045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:05.935873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:05.936173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:05.936914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:05.939118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:05.978530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 959/1931] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:32:05.981733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:06.010112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4a2a4a-f690-4c2a-9fe1-061828cd2552 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:06.015475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:06.016143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:06.016423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:06.016692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] PATH_INFO: `/v3/projects/0d0a8de3542947c2a3e99cb606aace3e/users/985faf33c9504b9c8bbf22286e80ad68/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:06.037058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:06.047689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 973/1932] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:06.051424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d0a8de3542947c2a3e99cb606aace3e, user_id=985faf33c9504b9c8bbf22286e80ad68, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:06.054854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:06.057068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935df92b-b601-4a24-9472-3253579e2abd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:06.058675 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-935df92b-b601-4a24-9472-3253579e2abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:06.090070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 960/1933] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:05 2022] PUT /identity/v3/projects/0d0a8de3542947c2a3e99cb606aace3e/users/985faf33c9504b9c8bbf22286e80ad68/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:06.091015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:06.184231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935df92b-b601-4a24-9472-3253579e2abd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:06.220021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907867fb-7f0b-43d0-8287-0f004f749430 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:06.231270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:06.231911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:06.232771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:06.233123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:06.233811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:06.240204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:06.263284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 961/1934] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:06 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:06.270399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:06.294167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4afc63-7a1a-404f-b68f-02d123e58f09 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:06.301286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a725bc6-567c-45a7-a761-3f53a14af284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:06.304419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a725bc6-567c-45a7-a761-3f53a14af284 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:06.304768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a725bc6-567c-45a7-a761-3f53a14af284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:06.305440 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a725bc6-567c-45a7-a761-3f53a14af284 admin admin] PATH_INFO: `/v3/projects/0d0a8de3542947c2a3e99cb606aace3e/users/985faf33c9504b9c8bbf22286e80ad68/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:06.308014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a725bc6-567c-45a7-a761-3f53a14af284 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d0a8de3542947c2a3e99cb606aace3e, user_id=985faf33c9504b9c8bbf22286e80ad68, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:06.312502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a725bc6-567c-45a7-a761-3f53a14af284 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:06.337159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 962/1935] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:06 2022] PUT /identity/v3/projects/0d0a8de3542947c2a3e99cb606aace3e/users/985faf33c9504b9c8bbf22286e80ad68/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:06.342722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:06.343141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:06.343478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:06.343828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:06.347157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:06.347773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:06.347852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:06.398235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] MFA Rules not processed for user `985faf33c9504b9c8bbf22286e80ad68`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:06.399830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:06.410508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:06.458845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:06.460546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:06.472403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 974/1936] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:32:06.594592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 963/1937] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:06 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:06.606720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:06.608584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:06.625697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292516a-8d2c-4679-9e36-c0a27413347d service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:06.754328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92c25e6-8120-4746-a46d-ab2a0beeaedf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:06.755120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c25e6-8120-4746-a46d-ab2a0beeaedf service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:06.755785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c25e6-8120-4746-a46d-ab2a0beeaedf service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:06.756157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c25e6-8120-4746-a46d-ab2a0beeaedf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:06.774416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92c25e6-8120-4746-a46d-ab2a0beeaedf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:06.776289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92c25e6-8120-4746-a46d-ab2a0beeaedf service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:06.798695 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 975/1938] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:06 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:07.472551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:07.553370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:07.561907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea3a26-8031-49ca-96a4-76511f1bbf15 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:07.571008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:07.571767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:07.572117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:07.572583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] PATH_INFO: `/v3/users/6567a4ca408845c4b25593071eea257d` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:07.575302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6567a4ca408845c4b25593071eea257d)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:07.577312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:07.647010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6567a4ca408845c4b25593071eea257d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:07.664362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6567a4ca408845c4b25593071eea257d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:07.665344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6567a4ca408845c4b25593071eea257d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:07.675079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6567a4ca408845c4b25593071eea257d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:07.675841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6567a4ca408845c4b25593071eea257d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:07.697092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 964/1939] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:07 2022] DELETE /identity/v3/users/6567a4ca408845c4b25593071eea257d => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:07.699108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316643f-3dfd-4006-9aca-0ac7a2b62bec None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:07.699610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316643f-3dfd-4006-9aca-0ac7a2b62bec None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:07.699715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316643f-3dfd-4006-9aca-0ac7a2b62bec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:07.722062 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-9316643f-3dfd-4006-9aca-0ac7a2b62bec None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:07.723300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9316643f-3dfd-4006-9aca-0ac7a2b62bec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:07.806196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 976/1940] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:07.816850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:07.818436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:07.910555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab53de3-8130-4bff-a04c-b2b20fab8f63 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:07.931021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:07.931910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:07.932136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:07.932411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:07.959596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:07.962733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:08.003194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 965/1941] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:08.268294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:08.268578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:08.268896 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:08.297230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] MFA Rules not processed for user `1fceec891ef24ba3a39da72c90fb733a`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:08.298470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:08.434435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 977/1942] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:08.815069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:08.817014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:08.839665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c180e7f-17eb-4609-834b-990d58a50688 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:08.873877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:08.874937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:08.875911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:08.876471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] PATH_INFO: `/v3/projects/052ec677934c4fe4a422d3c857f8ad44` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:08.878577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=052ec677934c4fe4a422d3c857f8ad44)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:08.903785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:09.008080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '052ec677934c4fe4a422d3c857f8ad44', 'request_id': 'req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.022587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '052ec677934c4fe4a422d3c857f8ad44', 'request_id': 'req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.036027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '052ec677934c4fe4a422d3c857f8ad44', 'request_id': 'req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.074540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:09.083551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] The token cache is being invalidate because project 052ec677934c4fe4a422d3c857f8ad44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:09.084249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.085681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.088335 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 966/1943] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:08 2022] DELETE /identity/v3/projects/052ec677934c4fe4a422d3c857f8ad44 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:09.093352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:09.106497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5d2e6e-bc59-41a0-9fa8-f5f5cbb41188 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:09.132458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:09.181575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:09.182321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:09.182803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:09.183193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] PATH_INFO: `/v3/users/985faf33c9504b9c8bbf22286e80ad68` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:09.184404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] RBAC: Authorizing `identity:delete_user(user_id=985faf33c9504b9c8bbf22286e80ad68)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:09.186978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:09.269204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '985faf33c9504b9c8bbf22286e80ad68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.301240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '985faf33c9504b9c8bbf22286e80ad68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.302032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '985faf33c9504b9c8bbf22286e80ad68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.314146 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '985faf33c9504b9c8bbf22286e80ad68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.315228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '985faf33c9504b9c8bbf22286e80ad68'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.331897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 978/1944] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:09 2022] DELETE /identity/v3/users/985faf33c9504b9c8bbf22286e80ad68 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:09.591913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:09.593379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:09.620657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330237cc-a7f0-44b2-88f0-9efb6ddbefe9 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:09.671341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:09.672931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:09.714947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:09.716181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:09.717043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:09.717319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] PATH_INFO: `/v3/users/2bff8ece33e9468593d74623841a8497` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:09.718908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bff8ece33e9468593d74623841a8497)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:09.722612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:09.781005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcbe2ff-b995-46b9-ac4c-8da34d22e847 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:09.787405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:09.788250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:09.788659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:09.788959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] PATH_INFO: `/v3/projects/0d0a8de3542947c2a3e99cb606aace3e` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:09.790426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d0a8de3542947c2a3e99cb606aace3e)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:09.792385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:09.796225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bff8ece33e9468593d74623841a8497'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.817105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bff8ece33e9468593d74623841a8497'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.817864 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bff8ece33e9468593d74623841a8497'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.830843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bff8ece33e9468593d74623841a8497'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.831651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-6e521c62-ee3f-4817-bb98-fb4dce256a99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bff8ece33e9468593d74623841a8497'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.851009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:09.866276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 967/1945] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:09 2022] DELETE /identity/v3/users/2bff8ece33e9468593d74623841a8497 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:09.868080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d0a8de3542947c2a3e99cb606aace3e', 'request_id': 'req-0b0addee-5703-436a-a2f2-d5d08408684a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.872340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:09.873118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:09.873387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:09.886981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d0a8de3542947c2a3e99cb606aace3e', 'request_id': 'req-0b0addee-5703-436a-a2f2-d5d08408684a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.902621 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:09.904422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d0a8de3542947c2a3e99cb606aace3e', 'request_id': 'req-0b0addee-5703-436a-a2f2-d5d08408684a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.911684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:09.922264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] The token cache is being invalidate because project 0d0a8de3542947c2a3e99cb606aace3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:09.922601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.923258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b0addee-5703-436a-a2f2-d5d08408684a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:09.924963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 979/1946] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:09 2022] DELETE /identity/v3/projects/0d0a8de3542947c2a3e99cb606aace3e => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:09.930289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91173d2a-75b4-4b0b-ad52-c1979d89571f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:09.930849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91173d2a-75b4-4b0b-ad52-c1979d89571f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:09.931162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91173d2a-75b4-4b0b-ad52-c1979d89571f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:09.956131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-91173d2a-75b4-4b0b-ad52-c1979d89571f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:09.957666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-91173d2a-75b4-4b0b-ad52-c1979d89571f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:09.989811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 968/1947] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:09.999072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.000492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.022019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 980/1948] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:10.030785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:10.031526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.031808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.032079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.069800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:10.070809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.079010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 981/1949] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:10.082897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.092003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b846435-2880-4043-ab01-ffcc734f8a5c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.101302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.101302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.102031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbb639-6207-4625-9826-c27fb1fa3101 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.102135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.102135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.102135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:32:10.102135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.102661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.115930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.117130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 982/1950] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:32:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:32:10.118442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.118885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.119521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.121349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.132446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.132815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c5fa2b-fc58-49c4-9fa5-3206535666f9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.134062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.137497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.138097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.138421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.138727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.139306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.141233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.144750 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 969/1951] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:32:10.174699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 983/1952] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:10.175898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.186836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086da3b-8128-4f20-a5eb-2061abe4e81f service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.247199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.248667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.248972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.249240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.249868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.251801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.273869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 970/1953] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:10.277601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.287929 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023d8f63-5e97-420b-b0a4-d4d42b1b47ff admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.292603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.293298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.293578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.293846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.295009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.296599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.309063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 984/1954] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:10.313852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.325463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd36f7-5ffa-4872-8a0c-d4ecc51d7889 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.330231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ab430d-135d-4ea6-b345-6b017120b399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.331119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ab430d-135d-4ea6-b345-6b017120b399 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.331401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ab430d-135d-4ea6-b345-6b017120b399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.331669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ab430d-135d-4ea6-b345-6b017120b399 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.332310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ab430d-135d-4ea6-b345-6b017120b399 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.333996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ab430d-135d-4ea6-b345-6b017120b399 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.345087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 971/1955] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:10.349352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.360148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adbaae3-4954-4b85-b1d1-ad73001d2e19 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.365399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.366068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.366344 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.366617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] PATH_INFO: `/v3/projects/c4805c3e2afd45e9a36a06cda0782d3c/users/77d51b66129e411b9ed90367db5a310e/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.384619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4805c3e2afd45e9a36a06cda0782d3c, user_id=77d51b66129e411b9ed90367db5a310e, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.387452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.401334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 985/1956] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:10 2022] PUT /identity/v3/projects/c4805c3e2afd45e9a36a06cda0782d3c/users/77d51b66129e411b9ed90367db5a310e/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:10.412933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.413216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.413487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.442418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] MFA Rules not processed for user `77d51b66129e411b9ed90367db5a310e`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:10.443641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.505583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.507280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.516760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 972/1957] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:10.530118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.576508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:10.577072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:10.606035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5e375d-c77d-4cc9-8b3e-1e0eb5447207 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.612835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.613542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.613824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.614100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] PATH_INFO: `/v3/projects/88d84ab9c0dc460085cec44e656f53cd` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.615446 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88d84ab9c0dc460085cec44e656f53cd)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.618028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.622407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c640b8f-6242-4f5e-8673-de451faf8525 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.628252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.629211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.629525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.629796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.630431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.633015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.665314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 973/1958] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/projects => generated 375 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:10.669342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.685224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88d84ab9c0dc460085cec44e656f53cd', 'request_id': 'req-3fc640a8-fa7a-4680-8352-18649c92c3a5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:10.690732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88d84ab9c0dc460085cec44e656f53cd', 'request_id': 'req-3fc640a8-fa7a-4680-8352-18649c92c3a5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:10.699056 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88d84ab9c0dc460085cec44e656f53cd', 'request_id': 'req-3fc640a8-fa7a-4680-8352-18649c92c3a5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:10.713200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] The token cache is being invalidate because project 88d84ab9c0dc460085cec44e656f53cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:10.713501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:10.714642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3fc640a8-fa7a-4680-8352-18649c92c3a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:10.719967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 986/1959] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:10 2022] DELETE /identity/v3/projects/88d84ab9c0dc460085cec44e656f53cd => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:10.721461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3d4ff-840f-4f5a-98d8-4525ce28b96b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.721726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3d4ff-840f-4f5a-98d8-4525ce28b96b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.721919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3d4ff-840f-4f5a-98d8-4525ce28b96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.743011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-bae3d4ff-840f-4f5a-98d8-4525ce28b96b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:10.744247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bae3d4ff-840f-4f5a-98d8-4525ce28b96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.748279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5941bf22-e4b2-4d73-b5fa-a6f58a7b75b6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.752472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 987/1960] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:10.757276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.757882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.758159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.758425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.759145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.760169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50068754-a23d-474f-96c3-a6a809689205 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.760455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50068754-a23d-474f-96c3-a6a809689205 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.760754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50068754-a23d-474f-96c3-a6a809689205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.761482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.781385 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-50068754-a23d-474f-96c3-a6a809689205 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:10.782579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-50068754-a23d-474f-96c3-a6a809689205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.787368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 974/1961] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/users => generated 288 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:10.788240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.789839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:10.791498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 988/1962] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:10.795261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50068754-a23d-474f-96c3-a6a809689205 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.804157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ec2fc-9fe0-4f76-b65b-ab62ad0b5025 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.805637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50068754-a23d-474f-96c3-a6a809689205 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.809277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.809876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.810179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.810278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.810447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.810865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.811106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.811210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.811397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.812035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:32:10.812335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.812723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.813947 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.824401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 975/1963] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:10.830162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 989/1964] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:32:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:32:10.832559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.833745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.842781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863197b1-b8c6-44ce-96c5-532ea457bb5f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.846130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ef8bd-579c-408b-af1b-cbb901d82b66 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.848216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.848849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.849149 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.849416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.850051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.851135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.851726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.851842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.852024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.852292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.852960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.855480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.864887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 976/1965] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:10.866669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.880004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaf24ad-651a-4366-8bdf-5446b6936e8d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:10.885495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed0f33b-a62d-4a68-94e0-d35b5a65ff3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:10.886014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed0f33b-a62d-4a68-94e0-d35b5a65ff3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.886289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed0f33b-a62d-4a68-94e0-d35b5a65ff3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.886552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed0f33b-a62d-4a68-94e0-d35b5a65ff3e admin admin] PATH_INFO: `/v3/projects/479c91d842e74a6fac193d4fbdfbc571/users/d6b53861bb3c427a92d7e900ce18856f/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.890600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 990/1966] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/projects => generated 369 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:10.892370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:10.913111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed0f33b-a62d-4a68-94e0-d35b5a65ff3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=479c91d842e74a6fac193d4fbdfbc571, user_id=d6b53861bb3c427a92d7e900ce18856f, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:10.919043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed0f33b-a62d-4a68-94e0-d35b5a65ff3e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:10.954870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 977/1967] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:10 2022] PUT /identity/v3/projects/479c91d842e74a6fac193d4fbdfbc571/users/d6b53861bb3c427a92d7e900ce18856f/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:10.967194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:10.967494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:10.969193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:10.976053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38abab-c55d-408a-bd4a-fde020d57cf0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.013890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] MFA Rules not processed for user `d6b53861bb3c427a92d7e900ce18856f`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:11.015757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:11.034106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:11.066242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.067168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.067597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.068040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.068982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.071789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.108905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 991/1968] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/users => generated 285 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:11.109994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.111546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:11.124933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c801f30-14cb-4331-93d0-67582f6a9f13 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.132027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.133019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.133389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.133775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.134640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.136968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.145203 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 978/1969] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:10 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:11.153409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.156369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 992/1970] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:11 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:11.159526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.168006 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b668d-1b19-41fc-901e-efe6b007880c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.171005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99de4a-4ff9-4169-a5e4-eeec513a5127 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.173233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.174062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.174563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.174991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.175775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.176019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.176690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.177070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.177408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.178111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.178740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.179865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.209305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 993/1971] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:11 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:11.210295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.222095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 979/1972] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:11 2022] POST /identity/v3/projects => generated 377 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:11.223799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.229810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1262af96-8094-4b6e-be2f-f14df6838d88 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.310435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f555f6-fb84-42f4-9755-358e3b7886a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.311261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f555f6-fb84-42f4-9755-358e3b7886a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.311543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f555f6-fb84-42f4-9755-358e3b7886a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.311812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f555f6-fb84-42f4-9755-358e3b7886a6 admin admin] PATH_INFO: `/v3/projects/1ed84c57878f41168d7e76eac8f33585/users/90bf148a7d8d446586513405bdba64f8/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.319824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90522b46-a073-4013-ba59-003cc2e4e626 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.326105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.326562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.328305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.328305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.328305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.330239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.344891 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f555f6-fb84-42f4-9755-358e3b7886a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ed84c57878f41168d7e76eac8f33585, user_id=90bf148a7d8d446586513405bdba64f8, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.348467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f555f6-fb84-42f4-9755-358e3b7886a6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.363675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 980/1973] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:11 2022] POST /identity/v3/users => generated 295 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:11.368647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.377820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 994/1974] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:11 2022] PUT /identity/v3/projects/1ed84c57878f41168d7e76eac8f33585/users/90bf148a7d8d446586513405bdba64f8/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:11.390186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.390719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.391171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.403720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bedd6c-f2a0-4c5c-aaa7-200ca2802670 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.442209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] MFA Rules not processed for user `90bf148a7d8d446586513405bdba64f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:11.444274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:11.525968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.526617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.530091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.530400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.531091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.535854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.562103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 981/1975] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:11 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:11.565887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.579755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 995/1976] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:11.593708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.595176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:11.595300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b863975-103b-4416-b461-2c2b64401cfb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.618891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.621267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.622549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.623025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] PATH_INFO: `/v3/projects/6577b69bff594ff5b77f24d3d537742c/users/bcca838aed5844c5bb2d1b925977685b/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.647266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:11.673762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6577b69bff594ff5b77f24d3d537742c, user_id=bcca838aed5844c5bb2d1b925977685b, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.676712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.706427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1740454-ee15-4b1f-96da-28f021fb9215 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.709823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 982/1977] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:11 2022] PUT /identity/v3/projects/6577b69bff594ff5b77f24d3d537742c/users/bcca838aed5844c5bb2d1b925977685b/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:11.714258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.772403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:11.825812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e60e589-7d7f-4b5c-a298-f3376e3ecbe0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.831556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.832260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.832799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.833084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.834011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.836582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.868833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.869816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.870021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.872129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.876597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 983/1978] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:11 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:11.877693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:11.904298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9658f44d-f4d3-4c43-8503-bedcf3edd527 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:11.925413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966febb1-496e-40b4-ab58-4ff0da66f23c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:11.925413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966febb1-496e-40b4-ab58-4ff0da66f23c admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.925413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966febb1-496e-40b4-ab58-4ff0da66f23c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.925413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966febb1-496e-40b4-ab58-4ff0da66f23c admin admin] PATH_INFO: `/v3/projects/6577b69bff594ff5b77f24d3d537742c/users/bcca838aed5844c5bb2d1b925977685b/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.927891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966febb1-496e-40b4-ab58-4ff0da66f23c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6577b69bff594ff5b77f24d3d537742c, user_id=bcca838aed5844c5bb2d1b925977685b, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.930827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966febb1-496e-40b4-ab58-4ff0da66f23c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.942179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 984/1979] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:11 2022] PUT /identity/v3/projects/6577b69bff594ff5b77f24d3d537742c/users/bcca838aed5844c5bb2d1b925977685b/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:11.952400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:11.952817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:11.953147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:11.963317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:11.966984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:11.981996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 996/1980] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:11 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:32:11.986665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] MFA Rules not processed for user `bcca838aed5844c5bb2d1b925977685b`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:11.988134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:12.095548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 985/1981] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:11 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:12.121365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:12.123566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:12.286990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4e1e0-5f9a-4358-922c-dca2a534e14d service glance] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:12.371956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:12.385003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:12.386898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:12.387176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:12.387438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:12.392936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:12.500852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:12.502412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6902b59-56fa-4df1-906b-503703f69f38 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:12.512960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 997/1982] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:12 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:14.198023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:14.199554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:14.212258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2863ee4-8213-4951-9fe4-8cbe8fc7b9ae None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:14.215863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:14.216511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:14.216821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:14.217095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:14.331166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:14.334353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:14.361720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 986/1983] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:14.751399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5d2c22-053a-4d28-b08f-6a54e3c19edf None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:14.751974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5d2c22-053a-4d28-b08f-6a54e3c19edf None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:14.752424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5d2c22-053a-4d28-b08f-6a54e3c19edf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:14.779682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1c5d2c22-053a-4d28-b08f-6a54e3c19edf None None] MFA Rules not processed for user `bcd7f1121dab4e8db4c747c51ed49240`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:14.781550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1c5d2c22-053a-4d28-b08f-6a54e3c19edf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:14.854554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 998/1984] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:14 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:15.238794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:15.247390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23649e5b-866b-45ac-b4de-feca7f669aec service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:15.251435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c580a98-e9d5-4e75-889a-c6f88d1e1c03 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:15.252167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c580a98-e9d5-4e75-889a-c6f88d1e1c03 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:15.252445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c580a98-e9d5-4e75-889a-c6f88d1e1c03 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:15.253150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c580a98-e9d5-4e75-889a-c6f88d1e1c03 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:15.262619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c580a98-e9d5-4e75-889a-c6f88d1e1c03 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:15.264776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c580a98-e9d5-4e75-889a-c6f88d1e1c03 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:15.280281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 987/1985] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:15 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:16.550430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c4d8c7-c460-4538-9cac-53c7703590ba None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:16.551073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c4d8c7-c460-4538-9cac-53c7703590ba None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:16.551073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c4d8c7-c460-4538-9cac-53c7703590ba None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:16.552659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 999/1986] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:32:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:32:16.559579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e0056-f47c-49fa-ba90-43be8525339b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:16.560131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e0056-f47c-49fa-ba90-43be8525339b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:16.560228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e0056-f47c-49fa-ba90-43be8525339b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:16.581043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b15e0056-f47c-49fa-ba90-43be8525339b None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:16.582613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b15e0056-f47c-49fa-ba90-43be8525339b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:16.604431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:16.668100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 988/1987] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:32:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:16.681656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c4d8c7-c460-4538-9cac-53c7703590ba None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:16.770211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c4d8c7-c460-4538-9cac-53c7703590ba None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:16.774902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b72675f-bf39-4f3b-819f-7487a35f5686 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:16.775754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b72675f-bf39-4f3b-819f-7487a35f5686 service glance] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:16.776178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b72675f-bf39-4f3b-819f-7487a35f5686 service glance] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:16.776650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b72675f-bf39-4f3b-819f-7487a35f5686 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:16.795569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b72675f-bf39-4f3b-819f-7487a35f5686 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:16.797394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b72675f-bf39-4f3b-819f-7487a35f5686 service glance] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:16.832606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1000/1988] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:32:16.919587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a341a-2f3d-4d5d-adbe-eb1c898ca94c None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:16.919875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a341a-2f3d-4d5d-adbe-eb1c898ca94c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:16.920171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a341a-2f3d-4d5d-adbe-eb1c898ca94c None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:16.921793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 989/1989] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:32:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:32:16.931357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e487c7-17b5-4327-a041-94d1ced613fd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:16.931634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e487c7-17b5-4327-a041-94d1ced613fd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:16.932070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e487c7-17b5-4327-a041-94d1ced613fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:16.958023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-78e487c7-17b5-4327-a041-94d1ced613fd None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:16.959222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-78e487c7-17b5-4327-a041-94d1ced613fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:17.003024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:17.009985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1001/1990] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:32:16 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:17.022166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a341a-2f3d-4d5d-adbe-eb1c898ca94c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:17.023830 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6a1a341a-2f3d-4d5d-adbe-eb1c898ca94c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:17.037316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a341a-2f3d-4d5d-adbe-eb1c898ca94c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:17.040847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c01e9e5-c97a-4cee-9045-ed4cdf385401 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:17.041437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c01e9e5-c97a-4cee-9045-ed4cdf385401 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:17.041709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c01e9e5-c97a-4cee-9045-ed4cdf385401 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:17.041978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c01e9e5-c97a-4cee-9045-ed4cdf385401 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:17.049078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c01e9e5-c97a-4cee-9045-ed4cdf385401 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:17.050516 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c01e9e5-c97a-4cee-9045-ed4cdf385401 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:17.058419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 990/1991] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:17.143289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47646f8a-f94d-45f0-8534-0e98cbad5535 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:17.143667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47646f8a-f94d-45f0-8534-0e98cbad5535 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:17.144155 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47646f8a-f94d-45f0-8534-0e98cbad5535 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:17.145338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1002/1992] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:32:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:32:17.150786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:17.151072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:17.151345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:17.171113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:17.172312 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:17.181537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 991/1993] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:32:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:17.192282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47646f8a-f94d-45f0-8534-0e98cbad5535 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:17.203735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47646f8a-f94d-45f0-8534-0e98cbad5535 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:17.207304 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd6721a-8d6b-4f49-9639-b14a7ad9a5ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:17.207899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6721a-8d6b-4f49-9639-b14a7ad9a5ec service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:17.208171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6721a-8d6b-4f49-9639-b14a7ad9a5ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:17.208437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6721a-8d6b-4f49-9639-b14a7ad9a5ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:17.218456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd6721a-8d6b-4f49-9639-b14a7ad9a5ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:17.219867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd6721a-8d6b-4f49-9639-b14a7ad9a5ec service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:17.230287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1003/1994] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:21.789684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:21.902373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5919051-7ab1-4c51-80cc-0efd26308d23 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:21.916811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:21.917710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:21.918004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:21.919042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] PATH_INFO: `/v3/users/77d51b66129e411b9ed90367db5a310e` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:21.922054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] RBAC: Authorizing `identity:delete_user(user_id=77d51b66129e411b9ed90367db5a310e)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:21.924076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:22.032089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77d51b66129e411b9ed90367db5a310e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:22.054388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77d51b66129e411b9ed90367db5a310e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:22.055325 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77d51b66129e411b9ed90367db5a310e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:22.075699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77d51b66129e411b9ed90367db5a310e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:22.076581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77d51b66129e411b9ed90367db5a310e'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:22.092593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 992/1995] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:21 2022] DELETE /identity/v3/users/77d51b66129e411b9ed90367db5a310e => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:22.102885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:22.102885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:22.102885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:22.134106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:22.135349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:22.202194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1004/1996] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:22.221338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:22.223868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:22.242585 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:22.340632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7346fde-42c5-4e53-bb46-8f93e847266a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:22.346772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:22.349685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:22.350113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:22.350478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:22.369694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:22.371581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:22.383655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 993/1997] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:22.953909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:22.955476 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:22.968517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7d6f9e-e41e-4662-ae86-bf7a82392a3c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:22.976198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:22.982191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:22.982191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:22.982191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] PATH_INFO: `/v3/projects/c4805c3e2afd45e9a36a06cda0782d3c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:22.984367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4805c3e2afd45e9a36a06cda0782d3c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:22.986786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:23.045772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4805c3e2afd45e9a36a06cda0782d3c', 'request_id': 'req-04d20380-0b7f-48b2-82d5-8a4a48dcd812'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.051213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4805c3e2afd45e9a36a06cda0782d3c', 'request_id': 'req-04d20380-0b7f-48b2-82d5-8a4a48dcd812'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.063161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4805c3e2afd45e9a36a06cda0782d3c', 'request_id': 'req-04d20380-0b7f-48b2-82d5-8a4a48dcd812'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.076919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] The token cache is being invalidate because project c4805c3e2afd45e9a36a06cda0782d3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:23.077210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.078116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.079427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1005/1998] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:22 2022] DELETE /identity/v3/projects/c4805c3e2afd45e9a36a06cda0782d3c => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:23.083578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:23.094002 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450923f7-c003-49e2-b780-0052a0f2c295 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:23.150846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:23.151472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:23.151749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:23.152019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] PATH_INFO: `/v3/users/d6b53861bb3c427a92d7e900ce18856f` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:23.153181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6b53861bb3c427a92d7e900ce18856f)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:23.155122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:23.201475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6b53861bb3c427a92d7e900ce18856f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.213398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6b53861bb3c427a92d7e900ce18856f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.214065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6b53861bb3c427a92d7e900ce18856f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.220779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6b53861bb3c427a92d7e900ce18856f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.221398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6b53861bb3c427a92d7e900ce18856f'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.233952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 994/1999] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:23 2022] DELETE /identity/v3/users/d6b53861bb3c427a92d7e900ce18856f => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:23.510243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:23.511726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:23.524664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d20380-0b7f-48b2-82d5-8a4a48dcd812 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:23.583514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:23.584077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:23.584351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:23.584619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] PATH_INFO: `/v3/projects/479c91d842e74a6fac193d4fbdfbc571` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:23.585767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] RBAC: Authorizing `identity:delete_project(project_id=479c91d842e74a6fac193d4fbdfbc571)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:23.587670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:23.599724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:23.638215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '479c91d842e74a6fac193d4fbdfbc571', 'request_id': 'req-6aab51e0-02c1-45b8-a2f1-d555deb91039'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.643906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '479c91d842e74a6fac193d4fbdfbc571', 'request_id': 'req-6aab51e0-02c1-45b8-a2f1-d555deb91039'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.651083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '479c91d842e74a6fac193d4fbdfbc571', 'request_id': 'req-6aab51e0-02c1-45b8-a2f1-d555deb91039'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.662116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] The token cache is being invalidate because project 479c91d842e74a6fac193d4fbdfbc571 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:23.662400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.663037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.664632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1006/2000] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:23 2022] DELETE /identity/v3/projects/479c91d842e74a6fac193d4fbdfbc571 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:23.669272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:23.670722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:23.683350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b1cfaa-6d1e-4bd9-b60b-dc0af024888d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:23.716763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:23.743175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:23.743987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:23.744271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:23.744606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] PATH_INFO: `/v3/users/bcca838aed5844c5bb2d1b925977685b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:23.745875 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcca838aed5844c5bb2d1b925977685b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:23.748185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:23.795137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcca838aed5844c5bb2d1b925977685b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.808478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcca838aed5844c5bb2d1b925977685b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.809309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcca838aed5844c5bb2d1b925977685b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.818938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcca838aed5844c5bb2d1b925977685b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.819693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a62a4899-9d3f-4462-ba44-bed53fb00c1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcca838aed5844c5bb2d1b925977685b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:23.833148 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 995/2001] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:23 2022] DELETE /identity/v3/users/bcca838aed5844c5bb2d1b925977685b => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:24.217784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:24.219210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:24.235497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aab51e0-02c1-45b8-a2f1-d555deb91039 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:24.294732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:24.295421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:24.295699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:24.295970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] PATH_INFO: `/v3/projects/6577b69bff594ff5b77f24d3d537742c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:24.297122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] RBAC: Authorizing `identity:delete_project(project_id=6577b69bff594ff5b77f24d3d537742c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:24.299041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:24.345342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6577b69bff594ff5b77f24d3d537742c', 'request_id': 'req-0989665e-f8fe-461e-931e-ce0349854ddc'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:24.350816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6577b69bff594ff5b77f24d3d537742c', 'request_id': 'req-0989665e-f8fe-461e-931e-ce0349854ddc'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:24.357927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6577b69bff594ff5b77f24d3d537742c', 'request_id': 'req-0989665e-f8fe-461e-931e-ce0349854ddc'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:24.372686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] The token cache is being invalidate because project 6577b69bff594ff5b77f24d3d537742c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:24.373001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:24.373687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:24.375180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1007/2002] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:24 2022] DELETE /identity/v3/projects/6577b69bff594ff5b77f24d3d537742c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:25.838853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faca0f3-a867-4278-9d48-5032b7d82407 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:25.839304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faca0f3-a867-4278-9d48-5032b7d82407 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:25.839393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faca0f3-a867-4278-9d48-5032b7d82407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:25.861032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1faca0f3-a867-4278-9d48-5032b7d82407 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:25.862429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1faca0f3-a867-4278-9d48-5032b7d82407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:25.922963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 996/2003] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:32:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:25.936381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:25.938037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:25.975242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:26.019899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0989665e-f8fe-461e-931e-ce0349854ddc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:26.025194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:26.025846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:26.026123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:26.026420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:26.035597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:26.037271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:26.047369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1008/2004] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:32:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:32:26.188355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:26.188635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:26.189043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:26.220239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] MFA Rules not processed for user `bbcc7dcbf32b4048879bbc77e0998b52`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:26.221787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:26.314401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 997/2005] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:35.515664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:35.517115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:35.609589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f0c953-155b-48eb-818a-877ad3411c87 service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:35.615030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:35.615646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:35.615924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:35.616196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:35.618044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:35.631642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:35.633872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddbefd-4f39-4498-b910-55ce620fc469 service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:35.645909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1009/2006] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:36.307811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:44.657068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:44.658515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:44.727702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2d89ea-ee21-41e5-a70e-41fa0bdc9b10 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:44.733383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:44.734309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:44.734606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:44.734949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] PATH_INFO: `/v3/users/90bf148a7d8d446586513405bdba64f8` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:44.736234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] RBAC: Authorizing `identity:delete_user(user_id=90bf148a7d8d446586513405bdba64f8)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:44.738871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:44.802778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90bf148a7d8d446586513405bdba64f8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:44.818166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90bf148a7d8d446586513405bdba64f8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:44.818963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90bf148a7d8d446586513405bdba64f8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:44.832088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90bf148a7d8d446586513405bdba64f8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:44.832794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90bf148a7d8d446586513405bdba64f8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:44.847760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 998/2007] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:32:44 2022] DELETE /identity/v3/users/90bf148a7d8d446586513405bdba64f8 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:44.852145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:44.852647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:44.852791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:44.873370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:44.874584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:44.939854 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1010/2008] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:44.951139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:44.952568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:45.028336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5098a619-2a20-4860-8883-10f7040bdd60 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.031939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4d741-aa79-4db8-bc3e-3c10a3914eb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.032535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4d741-aa79-4db8-bc3e-3c10a3914eb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.032836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4d741-aa79-4db8-bc3e-3c10a3914eb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.033104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4d741-aa79-4db8-bc3e-3c10a3914eb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.046104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4d741-aa79-4db8-bc3e-3c10a3914eb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.047524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4d741-aa79-4db8-bc3e-3c10a3914eb7 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.061464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 999/2009] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:45.344808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.346275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:45.358604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f18d69-4713-4a22-bed2-4284db4cb777 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.364664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.365348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.365643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.365904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] PATH_INFO: `/v3/projects/1ed84c57878f41168d7e76eac8f33585` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.367122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ed84c57878f41168d7e76eac8f33585)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.370168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.419175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:45.422761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ed84c57878f41168d7e76eac8f33585', 'request_id': 'req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:45.429473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ed84c57878f41168d7e76eac8f33585', 'request_id': 'req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:45.436977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ed84c57878f41168d7e76eac8f33585', 'request_id': 'req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:45.448426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] The token cache is being invalidate because project 1ed84c57878f41168d7e76eac8f33585 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:32:45.448736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:45.449912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b60f22e7-5ee1-4b50-a23b-d146e2a82e37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:32:45.451420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1011/2010] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:32:45 2022] DELETE /identity/v3/projects/1ed84c57878f41168d7e76eac8f33585 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:45.456476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.456789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.457064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.477299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:45.478499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:45.496247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:45.541782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1000/2011] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:45.552903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.552903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.552903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.571522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:45.572731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:45.580768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1012/2012] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:45.584652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.594690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3674e5-2966-48f7-aa08-5cbf5ab52f35 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.599469 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.600064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.600341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.600607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.601279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:32:45.601577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.603170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.616271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1001/2013] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:32:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:32:45.620918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.631423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57df6334-a15b-49fc-8477-f4726e9156d1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.636393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.637200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.637563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.637900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.638606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.640579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.673201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1013/2014] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:45 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:45.677431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.688521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a7e6f-e9c4-4455-b6d7-0277faa0b528 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.748407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.749048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.749327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.749592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.750237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.752155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.776735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1002/2015] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:45 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:45.780014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.791538 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eba00f-3c41-4cb8-9406-c4972b7554c1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.797183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.797789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.798064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.798332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.798969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.800602 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.811175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1014/2016] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:32:45.815150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.826309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dc8a32-a62c-4a8e-803a-1f1fd734d25b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.831612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b1ce0f-08ed-415e-9481-f43f413666bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.832303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b1ce0f-08ed-415e-9481-f43f413666bb admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.832669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b1ce0f-08ed-415e-9481-f43f413666bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.833024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b1ce0f-08ed-415e-9481-f43f413666bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.833698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b1ce0f-08ed-415e-9481-f43f413666bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.835653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b1ce0f-08ed-415e-9481-f43f413666bb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.847977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1003/2017] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:45.851969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:45.863070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b235d34d-d34c-40da-9e18-14defe5de626 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:45.867808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:45.868546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.868753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.869259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] PATH_INFO: `/v3/projects/c57365923b844fe9b29d9e223ad49ce4/users/945de18174894cd683ebf34372e845f0/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.887488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c57365923b844fe9b29d9e223ad49ce4, user_id=945de18174894cd683ebf34372e845f0, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:45.890610 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:45.910436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1015/2018] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:45 2022] PUT /identity/v3/projects/c57365923b844fe9b29d9e223ad49ce4/users/945de18174894cd683ebf34372e845f0/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:45.915513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4625370-6940-44d1-b287-535082f5a867 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:45.915798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4625370-6940-44d1-b287-535082f5a867 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:45.916186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4625370-6940-44d1-b287-535082f5a867 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:45.945064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-e4625370-6940-44d1-b287-535082f5a867 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:45.946426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e4625370-6940-44d1-b287-535082f5a867 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:46.018073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1004/2019] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:46.027906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:46.029669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:46.092029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba59ac-b0fc-4983-ab19-3d9689dfa92c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:46.096301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:46.097040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:46.097421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:46.097693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:46.109069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:46.110416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:46.121784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1016/2020] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:32:50.795272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:50.795833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:50.795833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:50.837557 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] MFA Rules not processed for user `945de18174894cd683ebf34372e845f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:50.839413 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:50.865143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:50.972745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1005/2021] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:32:50.976190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:50.986125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088d51ea-ba67-4499-a06e-002be344fc47 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:50.992184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:50.992906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:50.993287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:50.993637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:50.994491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:50.996953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:51.027546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1017/2022] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:32:50 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:32:51.031532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:51.042738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eb612b-9a28-4e86-a148-40cfef0089ff None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:51.106639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:51.107263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:51.107592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:51.107867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:51.108522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:51.110510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:51.135293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1006/2023] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:32:51 2022] POST /identity/v3/users => generated 298 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:32:51.139875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:51.150477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed2f96c-fc7e-44f4-9023-9789d92639b4 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:51.155435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:51.156052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:51.156329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:51.156597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:51.157281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:51.159579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:51.171100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1018/2024] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:51.175465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:51.187609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6996067-6e80-485b-b2c8-0feea92d37b4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:51.192559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:51.193331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:51.193612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:51.193903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] PATH_INFO: `/v3/projects/7dc12b5da7a147b0a65eec8d04802cd2/users/5f09d9a29b0c41c48408fef8affc54bd/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:51.212897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dc12b5da7a147b0a65eec8d04802cd2, user_id=5f09d9a29b0c41c48408fef8affc54bd, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:51.215722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:51.232910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1007/2025] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:51 2022] PUT /identity/v3/projects/7dc12b5da7a147b0a65eec8d04802cd2/users/5f09d9a29b0c41c48408fef8affc54bd/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:32:51.238766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:51.265692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f71a52-d584-4bdb-bbd5-db294a844f98 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:51.312837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:32:51.338774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4344664c-9105-4dae-b791-c316addb70aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:51.339477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4344664c-9105-4dae-b791-c316addb70aa admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:51.339760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4344664c-9105-4dae-b791-c316addb70aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:51.340034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4344664c-9105-4dae-b791-c316addb70aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:51.340730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4344664c-9105-4dae-b791-c316addb70aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:51.342422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4344664c-9105-4dae-b791-c316addb70aa admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:51.353446 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1019/2026] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:32:51 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:32:51.358163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:51.369662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b56f21-7496-4709-9955-8b321226592e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:32:51.376881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:32:51.377776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:51.378218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:51.378487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] PATH_INFO: `/v3/projects/7dc12b5da7a147b0a65eec8d04802cd2/users/5f09d9a29b0c41c48408fef8affc54bd/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:51.380458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dc12b5da7a147b0a65eec8d04802cd2, user_id=5f09d9a29b0c41c48408fef8affc54bd, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:32:51.383280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:32:51.395087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1008/2027] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:32:51 2022] PUT /identity/v3/projects/7dc12b5da7a147b0a65eec8d04802cd2/users/5f09d9a29b0c41c48408fef8affc54bd/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:32:59.680169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202a3f03-b3f8-4dd9-93fc-ac6a85df7d8a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:32:59.681253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202a3f03-b3f8-4dd9-93fc-ac6a85df7d8a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:32:59.681679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202a3f03-b3f8-4dd9-93fc-ac6a85df7d8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:32:59.743443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-202a3f03-b3f8-4dd9-93fc-ac6a85df7d8a None None] MFA Rules not processed for user `5f09d9a29b0c41c48408fef8affc54bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:32:59.745010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-202a3f03-b3f8-4dd9-93fc-ac6a85df7d8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:32:59.931467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1020/2028] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:32:59 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:32:59.953537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:32:59.955345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:00.033191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641790b3-2d93-4ee4-a979-533be894d1c7 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:00.038204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae183699-1514-4fe3-9302-3675d5ca03d3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:00.038824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae183699-1514-4fe3-9302-3675d5ca03d3 service glance] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:00.039098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae183699-1514-4fe3-9302-3675d5ca03d3 service glance] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:00.040047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae183699-1514-4fe3-9302-3675d5ca03d3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:00.074792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:00.137225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae183699-1514-4fe3-9302-3675d5ca03d3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:00.138702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae183699-1514-4fe3-9302-3675d5ca03d3 service glance] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:00.150493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1009/2029] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:32:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:33:00.330268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:00.330713 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:00.331091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:00.355491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:00.361976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:00.421401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:00.421656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:00.421933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:00.467444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:00.470454 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:00.511447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1021/2030] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:33:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:33:00.548472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:00.550860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:00.628301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1010/2031] 10.222.0.23 () {58 vars in 967 bytes} [Wed Oct 12 02:33:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:33:00.645333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:00.646981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:00.704566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b9cfde-52c2-4b91-883e-e55f071f2bd0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:00.721005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18045ce3-5043-4c2f-adb4-e19ab4f4f527 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:00.721806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18045ce3-5043-4c2f-adb4-e19ab4f4f527 service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:00.722179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18045ce3-5043-4c2f-adb4-e19ab4f4f527 service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:00.722544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18045ce3-5043-4c2f-adb4-e19ab4f4f527 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:00.739114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:00.747079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18045ce3-5043-4c2f-adb4-e19ab4f4f527 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:00.753185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18045ce3-5043-4c2f-adb4-e19ab4f4f527 service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:00.775173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1022/2032] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:33:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:33:00.895494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c957d921-3eee-4e08-936c-2a3c102092c5 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:00.927042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e695ea1-9ee7-43da-91cb-faa822089efe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:00.927544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e695ea1-9ee7-43da-91cb-faa822089efe service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:00.927829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e695ea1-9ee7-43da-91cb-faa822089efe service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:00.928099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e695ea1-9ee7-43da-91cb-faa822089efe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:00.979056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e695ea1-9ee7-43da-91cb-faa822089efe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:00.984024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e695ea1-9ee7-43da-91cb-faa822089efe service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:01.014364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1011/2033] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:33:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:33:09.761018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:09.761560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:09.761560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:09.780939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] MFA Rules not processed for user `945de18174894cd683ebf34372e845f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:09.782170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:09.789016 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1023/2034] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:33:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:33:11.009057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=73 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:13.133167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:13.133741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:13.133741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:13.154654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:13.155885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:13.164331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1012/2035] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:33:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:33:13.177306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:13.189303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425a6561-7033-4736-8e92-5e5ea86cf1f9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:13.194622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:13.195248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:13.195526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:13.195796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:13.205170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:13.206832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:13.215938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1024/2036] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:33:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:33:23.148825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:23.211967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=11 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:46.657379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:46.753267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4352ec-4dd6-46e4-9bb1-e3cec8c396be None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:46.758523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:46.759142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:46.759781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:46.760062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:46.773227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:46.774971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:46.807685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1013/2037] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:33:46 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:33:56.454353 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:56.455856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:56.520619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5976c-b196-4252-a61c-abe4ebfd96df service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:56.524973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:56.525594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:56.525866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:56.526131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] PATH_INFO: `/v3/users/bbcc7dcbf32b4048879bbc77e0998b52` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:56.527855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbcc7dcbf32b4048879bbc77e0998b52)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:56.529847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:56.576386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbcc7dcbf32b4048879bbc77e0998b52'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.587839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbcc7dcbf32b4048879bbc77e0998b52'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.588500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbcc7dcbf32b4048879bbc77e0998b52'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.596894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbcc7dcbf32b4048879bbc77e0998b52'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.597671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c8ac2c33-7dab-40a7-be23-cc5b4688df0a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbcc7dcbf32b4048879bbc77e0998b52'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.610158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1025/2038] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:33:56 2022] DELETE /identity/v3/users/bbcc7dcbf32b4048879bbc77e0998b52 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:33:56.782491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:56.783854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=26 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:56.795760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1896651a-0006-48e9-a80c-d5fe092b0c3d service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:56.866403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:56.867047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:56.867351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:56.867772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] PATH_INFO: `/v3/projects/81144c673e7b44dcb3b0f700ee628719` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:56.868817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81144c673e7b44dcb3b0f700ee628719)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:56.871385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:56.932721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81144c673e7b44dcb3b0f700ee628719', 'request_id': 'req-82a39f03-ee6c-4868-9053-10e950624c47'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.939069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81144c673e7b44dcb3b0f700ee628719', 'request_id': 'req-82a39f03-ee6c-4868-9053-10e950624c47'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.945491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81144c673e7b44dcb3b0f700ee628719', 'request_id': 'req-82a39f03-ee6c-4868-9053-10e950624c47'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.958729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] The token cache is being invalidate because project 81144c673e7b44dcb3b0f700ee628719 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:33:56.959079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.959843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-82a39f03-ee6c-4868-9053-10e950624c47 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:33:56.961374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1014/2039] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:33:56 2022] DELETE /identity/v3/projects/81144c673e7b44dcb3b0f700ee628719 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:33:56.966371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:56.966653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:56.967005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:56.986885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:56.988074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:57.050620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1026/2040] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:33:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:33:57.117686 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.117823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.118095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.143517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:57.144845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:57.153393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1015/2041] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:33:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:33:57.157705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.168878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2344bbb9-dd4b-40ee-a3c2-05153fbc065e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.181566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.182198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.182475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.182743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.184075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:33:57.184384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.186010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.204788 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1027/2042] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:33:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:33:57.208544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.218522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b89cef6-3325-4ff7-83d4-4eaab206fd94 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.224246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.225004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.225257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.225446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.226093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.228071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.265251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1016/2043] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:33:57 2022] POST /identity/v3/projects => generated 397 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:33:57.269784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.284620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dddfa9a-3d43-46ba-ac22-30950b3dc496 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.289054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:57.351461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.352190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.352495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.352793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.353770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.356466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.387010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1028/2044] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:33:57 2022] POST /identity/v3/users => generated 299 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:33:57.387553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.401976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c383bdf5-0d6d-444d-ac47-c683db5e41f4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.408528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.409311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.409754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.410027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.410679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.413084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.432479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1017/2045] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:33:57 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:33:57.437359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.448950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3310ff9-cae7-4d7b-852e-225b143387c1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.454488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf11cf-7045-4047-bea8-35c5563f85b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.455145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf11cf-7045-4047-bea8-35c5563f85b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.455369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf11cf-7045-4047-bea8-35c5563f85b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.455633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf11cf-7045-4047-bea8-35c5563f85b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.456264 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bf11cf-7045-4047-bea8-35c5563f85b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.457983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bf11cf-7045-4047-bea8-35c5563f85b5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.469700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1029/2046] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:33:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:33:57.473825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.485375 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06488e35-53e8-4d81-b3d0-8f1139a8da46 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.490085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.490737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.491146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.491418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] PATH_INFO: `/v3/projects/3a65d140f7d54509a928d0ce891cebf2/users/fdf9edfb51ee4a25b03052b1a07712da/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.507068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a65d140f7d54509a928d0ce891cebf2, user_id=fdf9edfb51ee4a25b03052b1a07712da, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.509808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.524077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1018/2047] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:33:57 2022] PUT /identity/v3/projects/3a65d140f7d54509a928d0ce891cebf2/users/fdf9edfb51ee4a25b03052b1a07712da/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:33:57.535099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.535382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.535687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.564117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] MFA Rules not processed for user `fdf9edfb51ee4a25b03052b1a07712da`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:33:57.565411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:57.635946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1030/2048] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:33:57 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:33:57.648588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:33:57.650422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:33:57.682111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:33:57.710427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbae38-7507-4b26-92cc-83e30c45247a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:33:57.714069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082e87a-4e0b-49b3-a36f-297e6b0545c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:33:57.715173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082e87a-4e0b-49b3-a36f-297e6b0545c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:33:57.715444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082e87a-4e0b-49b3-a36f-297e6b0545c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:33:57.715708 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082e87a-4e0b-49b3-a36f-297e6b0545c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:33:57.724205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082e87a-4e0b-49b3-a36f-297e6b0545c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:33:57.725718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082e87a-4e0b-49b3-a36f-297e6b0545c6 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:33:57.736020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1019/2049] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:33:57 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:34:05.939324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:34:05.942243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31fffe9-07ac-4a65-a273-958e0790f4d8 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:05.943494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31fffe9-07ac-4a65-a273-958e0790f4d8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:05.943767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31fffe9-07ac-4a65-a273-958e0790f4d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:05.968243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c31fffe9-07ac-4a65-a273-958e0790f4d8 None None] MFA Rules not processed for user `1fceec891ef24ba3a39da72c90fb733a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:34:05.970216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c31fffe9-07ac-4a65-a273-958e0790f4d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:06.016519 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551d6de-d663-4192-9ba3-6829fe936651 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:34:06.021986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:34:06.022906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:06.023182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:06.023454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] PATH_INFO: `/v3/users/fdf9edfb51ee4a25b03052b1a07712da` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:06.025152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdf9edfb51ee4a25b03052b1a07712da)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:34:06.027667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:34:06.036735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1020/2050] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:34:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:34:06.047942 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:34:06.086510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdf9edfb51ee4a25b03052b1a07712da'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:06.121151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdf9edfb51ee4a25b03052b1a07712da'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:06.122063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdf9edfb51ee4a25b03052b1a07712da'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:06.130542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdf9edfb51ee4a25b03052b1a07712da'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:06.131315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdf9edfb51ee4a25b03052b1a07712da'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:06.145276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1031/2051] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:34:05 2022] DELETE /identity/v3/users/fdf9edfb51ee4a25b03052b1a07712da => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:34:06.149643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:06.149925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:06.150192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:06.171336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:34:06.172533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:06.236428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1021/2052] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:34:06.246063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:34:06.248823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:06.311550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f9d0e3-d3b2-4cad-9aca-0512ef01cf55 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:34:06.315190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a58b68-f976-48af-98a1-2e7725b8abee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:34:06.315791 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a58b68-f976-48af-98a1-2e7725b8abee service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:06.316061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a58b68-f976-48af-98a1-2e7725b8abee service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:06.316346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a58b68-f976-48af-98a1-2e7725b8abee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:06.327670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a58b68-f976-48af-98a1-2e7725b8abee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:34:06.328847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a58b68-f976-48af-98a1-2e7725b8abee service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:34:06.340441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1032/2053] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:34:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:34:06.899534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:34:06.901322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:06.914210 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2b28e6-fec7-4424-ab03-7636c00e1522 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:34:06.919631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:34:06.920260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:06.920542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:06.920851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] PATH_INFO: `/v3/projects/3a65d140f7d54509a928d0ce891cebf2` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:06.922117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a65d140f7d54509a928d0ce891cebf2)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:34:06.924420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:34:07.011443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a65d140f7d54509a928d0ce891cebf2', 'request_id': 'req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:07.018529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a65d140f7d54509a928d0ce891cebf2', 'request_id': 'req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:07.036740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a65d140f7d54509a928d0ce891cebf2', 'request_id': 'req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:07.054765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:34:07.064919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] The token cache is being invalidate because project 3a65d140f7d54509a928d0ce891cebf2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:34:07.065816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:07.067118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-01d99dbc-f78b-4e8a-a4de-91d68ab76cf4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:34:07.069251 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1022/2054] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:34:06 2022] DELETE /identity/v3/projects/3a65d140f7d54509a928d0ce891cebf2 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:34:07.447427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:07.447925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:07.448060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:07.467633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] MFA Rules not processed for user `43f16ca13925411fb76a330b6c95395d`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:34:07.469124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:07.527129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1033/2055] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:34:07 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:34:17.067181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:34:17.519800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=9,keystone:SELECT=71,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:34:18.315428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:18.315939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:18.316062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:18.336843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] MFA Rules not processed for user `bcd7f1121dab4e8db4c747c51ed49240`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:34:18.337976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:18.399226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1023/2056] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:34:18 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:34:19.818861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:34:19.820297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:19.878165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ddb7e5-0f9d-47a1-8afa-bb8bd1b589fd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:34:19.881799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:34:19.882411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:19.882687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:19.882957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:19.884897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:19.963416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:34:19.965457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:34:19.976933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1034/2057] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:34:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:34:21.706039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:34:21.708577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:21.722123 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b393a6-8cf7-44cc-a2e0-2a544b8416a3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:34:21.725781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:34:21.726403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:21.726680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:21.726953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:21.728824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:21.743915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:34:21.745921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:34:21.757591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1024/2058] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:34:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:34:29.977180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=54 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:34:31.756003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=31 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:34:58.875103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:34:58.876544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:58.890032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73b6c3-736c-4046-91a9-09b677f283f3 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:34:58.895498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:34:58.896088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:34:58.896361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:34:58.896627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:34:58.898313 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:34:58.954715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:34:58.956311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:34:58.965700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1035/2059] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:34:58 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:02.948156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:02.949579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:03.005278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad8c9d3-dfd2-4f45-b2ab-225c51cffdf5 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:03.008785 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:03.009379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:03.009656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:03.009925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:03.011639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:03.022226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:03.023641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:03.033044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1025/2060] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:02 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:05.452604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:05.463615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7593738a-bb84-4bb5-ac05-9bfbcac6fed8 service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:05.469150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:05.469787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:05.470068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:05.470339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:05.472078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:05.482917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:05.484755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dffb7d3-dde6-411d-b86b-78301caeb8e1 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:05.494334 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1036/2061] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:05 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:13.028953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=26 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:13.590675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:13.590675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:13.610074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004c41b-bf0d-4834-b743-3fbf92b664dd service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:13.616551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:13.617283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:13.617567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:13.617837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:13.619711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:13.636356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:13.638753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:13.651909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1026/2062] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:15.489529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=34 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:19.135351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:19.135916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:19.135916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:19.155824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] MFA Rules not processed for user `1fceec891ef24ba3a39da72c90fb733a`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:19.157054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:19.163783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1037/2063] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:19.466038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:19.475369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5569e-3316-490c-95d6-51465f91215c service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:19.479431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:19.480020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:19.480298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:19.480567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:19.482296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:19.491766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:19.493225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:19.501452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1027/2064] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:19.818721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:19.820518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:19.903163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105bf330-2ba2-47a8-9b10-8ff5b57406f1 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:19.908085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:19.908707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:19.908992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:19.909261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:19.911013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:19.922598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:19.924165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:19.934923 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1038/2065] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:22.040855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:22.050986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9093f5aa-e641-4c62-bb3c-b0ae39e1dc25 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:22.054854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:22.055461 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:22.056142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:22.056573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:22.062486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:22.073853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:22.075630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba12e87-6e6c-4159-8979-7829d30aa14f service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:22.088309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1028/2066] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:24.077311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:24.113082 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578bcec2-6159-4138-b59b-ffee68221ce3 service placement] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:24.121808 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:24.122978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:24.123708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:24.124218 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:24.126128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:24.139682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:24.141306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:24.153887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1039/2067] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:28.170305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:28.170816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:28.170925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:28.190881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:28.192178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:28.249283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1029/2068] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:35:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:28.291002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:28.292399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:28.360408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f7d6a-b475-4992-8aff-62ad9d8ee2ac service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:28.366383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:28.366993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:28.367266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:28.367533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:28.376187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:28.377798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:28.386280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1040/2069] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:35:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:35:28.717721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:28.732041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cd2056-29da-4811-a7bd-534cef5cdcc7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:28.736822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:28.737434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:28.737711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:28.737976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:28.739922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:28.831600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:28.833103 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:28.844769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1030/2070] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:29.241363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:29.264201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffacc5da-587f-4a27-a9b2-29c5b6207df0 service swift] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:29.277469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:29.278528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:29.279046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:29.279539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:29.281636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:29.306972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:29.315703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:29.340024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1041/2071] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:34.666988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:34.666988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:34.706806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5457d4be-961a-49ca-b3c6-aeb1f676fc47 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:34.720876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:34.723654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:34.724662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:34.728591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:34.733111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:34.780488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:34.780488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ae4869-c488-45e0-8b54-17e8d763aa78 service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:34.805642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:34.808492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:34.836243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1031/2072] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:34.859298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e012731-f016-4ee7-8903-7bd8453f6a32 service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:34.868576 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:34.881708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:34.881983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:34.886787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] PATH_INFO: `/v3/users/bcd7f1121dab4e8db4c747c51ed49240` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:34.894173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcd7f1121dab4e8db4c747c51ed49240)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:34.901109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:34.931320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:34.931618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:34.937553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:34.991424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:34.992659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:34.993115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:35.003288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcd7f1121dab4e8db4c747c51ed49240'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.008490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1032/2073] 10.222.0.23 () {58 vars in 967 bytes} [Wed Oct 12 02:35:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:35.030484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcd7f1121dab4e8db4c747c51ed49240'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.032193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcd7f1121dab4e8db4c747c51ed49240'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.039501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:35.049241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcd7f1121dab4e8db4c747c51ed49240'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.050490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcd7f1121dab4e8db4c747c51ed49240'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.066897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe593b3d-ff1b-4429-8ff7-6c056a7277c7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:35.080799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1042/2074] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:35:34 2022] DELETE /identity/v3/users/bcd7f1121dab4e8db4c747c51ed49240 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:35.089174 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:35.089824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:35.090096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:35.090394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:35.092309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:35.121173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:35.122497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f63c8a-cf2f-4ee7-baf1-308081b40c65 service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:35.132951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1033/2075] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:35:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:35:35.581108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:35.582853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:35.716040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99212fe7-43b4-4528-ad2e-e9826c883339 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:35.725928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:35.727386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:35.728964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:35.729498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] PATH_INFO: `/v3/projects/bd41b04d606949edbc9a3a2f88f991b2` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:35.730998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd41b04d606949edbc9a3a2f88f991b2)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:35.733504 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:35.841288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd41b04d606949edbc9a3a2f88f991b2', 'request_id': 'req-f6ec2f73-d2e1-4f00-9094-425685362ba4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.854599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd41b04d606949edbc9a3a2f88f991b2', 'request_id': 'req-f6ec2f73-d2e1-4f00-9094-425685362ba4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.866467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd41b04d606949edbc9a3a2f88f991b2', 'request_id': 'req-f6ec2f73-d2e1-4f00-9094-425685362ba4'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.891632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] The token cache is being invalidate because project bd41b04d606949edbc9a3a2f88f991b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:35:35.892124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.893852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f6ec2f73-d2e1-4f00-9094-425685362ba4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:35.899707 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1043/2076] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:35:35 2022] DELETE /identity/v3/projects/bd41b04d606949edbc9a3a2f88f991b2 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:35.901383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:35.901885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:35.902083 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:35.929628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:35.931624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:35.960066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:36.019255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1034/2077] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:36.027959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.027959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.028240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.075114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:36.076347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:36.101777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1044/2078] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:36.101957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:36.132781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971eb587-0c79-4e2d-89e5-1fd762f1b53b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:36.145734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:36.146387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.146663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.146931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.147613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:35:36.147916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:36.150368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:36.183880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1035/2079] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:35:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:35:36.185711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:36.203272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575c8449-a77a-4dd9-a662-1082bbce6409 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:36.215595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:36.216235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.216515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.216800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.217691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:36.219372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:36.284097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1045/2080] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:35:36 2022] POST /identity/v3/projects => generated 357 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:35:36.285373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:36.298337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117573c-7582-48f2-a442-eed05eda3bc5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:36.427452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:36.428267 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.429791 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.430859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.432483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:36.435851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:36.472283 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:36.475444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1036/2081] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:35:36 2022] POST /identity/v3/users => generated 279 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:35:36.497956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742112b5-6944-4516-98d0-303b5b1d94a8 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:36.508663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:36.510726 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.511216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.511643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.512643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:36.514879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:36.527386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1046/2082] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:36.531100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:36.543258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d63aca-dba8-44dd-9622-2b06e0a7b33a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:36.548748 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12eb149-8baa-4f7b-abcb-dae758739e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:36.549383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12eb149-8baa-4f7b-abcb-dae758739e23 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.549662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12eb149-8baa-4f7b-abcb-dae758739e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.549932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12eb149-8baa-4f7b-abcb-dae758739e23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.550777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12eb149-8baa-4f7b-abcb-dae758739e23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:36.554644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12eb149-8baa-4f7b-abcb-dae758739e23 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:36.572916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1037/2083] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:36.577672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:36.601566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514728d-23a9-477d-9157-5cf3af553f84 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:36.613945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:36.614945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.615884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.616343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] PATH_INFO: `/v3/projects/2e7dfa954e634c46b53e71b39dde73ac/users/28b883e86f6e44688767b09cd661c7b1/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.623404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:36.644025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e7dfa954e634c46b53e71b39dde73ac, user_id=28b883e86f6e44688767b09cd661c7b1, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:36.646744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:36.676739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1047/2084] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:35:36 2022] PUT /identity/v3/projects/2e7dfa954e634c46b53e71b39dde73ac/users/28b883e86f6e44688767b09cd661c7b1/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:36.687260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:36.687765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:36.688188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:36.745794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] MFA Rules not processed for user `28b883e86f6e44688767b09cd661c7b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:36.747761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:36.893757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1038/2085] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:36 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:36.894569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.027133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a275832-695e-4824-8f77-d2c9bd7a9fe0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:37.040120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:37.041445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.042214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.042214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.042928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:37.045171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:37.104388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1048/2086] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:35:36 2022] POST /identity/v3/projects => generated 355 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:35:37.105263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.124124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2099f3b-b8ec-42b0-8b93-c547100b3af9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:37.194829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:37.296560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:37.297716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.298326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.298959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.299734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:37.302201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:37.359552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.359782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1039/2087] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:35:37 2022] POST /identity/v3/users => generated 284 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:35:37.376133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596db6fd-68d3-4f88-95df-03345fa0498e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:37.382551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:37.383414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.383755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.384157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.384932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:37.386624 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:37.412252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1049/2088] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:37.417117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.429535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfbb861-5b2d-4ce1-b718-e9e5cdf76991 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:37.445902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:37.446628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.446903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.451545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] PATH_INFO: `/v3/projects/5842ddf8c54546b18ae0b06eb2e78b40/users/b934dd26d5a042b2b42d1e415b003fbc/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.472450 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5842ddf8c54546b18ae0b06eb2e78b40, user_id=b934dd26d5a042b2b42d1e415b003fbc, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:37.475282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:37.494035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1040/2089] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:35:37 2022] PUT /identity/v3/projects/5842ddf8c54546b18ae0b06eb2e78b40/users/b934dd26d5a042b2b42d1e415b003fbc/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:37.498541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.512106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e2166b-d650-4d89-b3be-9c7cb36b51f5 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:37.583771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa82b29-8937-445c-b93b-ac1e254eba97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:37.584547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa82b29-8937-445c-b93b-ac1e254eba97 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.584961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa82b29-8937-445c-b93b-ac1e254eba97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.585330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa82b29-8937-445c-b93b-ac1e254eba97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.588729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa82b29-8937-445c-b93b-ac1e254eba97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:37.588729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa82b29-8937-445c-b93b-ac1e254eba97 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:37.610287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.610858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1050/2090] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:37.621491 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3f35bd-ecd5-48a9-b211-74e707dffb5a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:37.629986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:37.630824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.631204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.631576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] PATH_INFO: `/v3/projects/5842ddf8c54546b18ae0b06eb2e78b40/users/b934dd26d5a042b2b42d1e415b003fbc/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.635623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5842ddf8c54546b18ae0b06eb2e78b40, user_id=b934dd26d5a042b2b42d1e415b003fbc, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:37.639000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:37.661371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1041/2091] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:35:37 2022] PUT /identity/v3/projects/5842ddf8c54546b18ae0b06eb2e78b40/users/b934dd26d5a042b2b42d1e415b003fbc/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:37.664641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:37.664968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:37.665273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:37.702482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] MFA Rules not processed for user `b934dd26d5a042b2b42d1e415b003fbc`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:37.703717 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:37.822970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:37.879059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1051/2092] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:37 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:37.887784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:37.889885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:38.021523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa88d49-ac20-4eff-a4cf-ed5ec900e85a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:38.027210 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:38.028079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:38.028539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:38.029037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:38.053584 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:38.060639 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:38.092618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1042/2093] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:37 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:38.212957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:38.227416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2780c95-d005-44e9-9505-99cf52acad94 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:38.231541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:38.232223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:38.232802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:38.233080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:38.334525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:38.337010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:38.372495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1052/2094] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:38 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:44.536554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:44.538161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:44.620548 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:44.622809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:44.648145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a320922-e680-4c34-b94e-b15ee52d3779 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:44.655390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:44.657146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:44.657762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:44.660289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:44.664383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:44.678632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:44.680299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:44.690849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1043/2095] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:44.701090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60499da-942f-4804-9a2b-a21229170513 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:44.705973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:44.706601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:44.706876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:44.707141 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] PATH_INFO: `/v3/users/945de18174894cd683ebf34372e845f0` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:44.708470 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=945de18174894cd683ebf34372e845f0)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:44.710494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:44.757795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '945de18174894cd683ebf34372e845f0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:44.769430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '945de18174894cd683ebf34372e845f0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:44.770073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '945de18174894cd683ebf34372e845f0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:44.777617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '945de18174894cd683ebf34372e845f0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:44.778235 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '945de18174894cd683ebf34372e845f0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:44.785828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:44.791852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1053/2096] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:35:44 2022] DELETE /identity/v3/users/945de18174894cd683ebf34372e845f0 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:44.965857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:44.977218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e30737c-6ea9-411b-b96e-78476ef3c065 service placement] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:44.981425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:45.045190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:45.046028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.049093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.049549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] PATH_INFO: `/v3/projects/c57365923b844fe9b29d9e223ad49ce4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.052910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c57365923b844fe9b29d9e223ad49ce4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:45.056095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:45.128450 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c57365923b844fe9b29d9e223ad49ce4', 'request_id': 'req-c2ee0159-949c-4a55-bd88-825a2b2e0186'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.135336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c57365923b844fe9b29d9e223ad49ce4', 'request_id': 'req-c2ee0159-949c-4a55-bd88-825a2b2e0186'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.142628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c57365923b844fe9b29d9e223ad49ce4', 'request_id': 'req-c2ee0159-949c-4a55-bd88-825a2b2e0186'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.156960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] The token cache is being invalidate because project c57365923b844fe9b29d9e223ad49ce4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:35:45.157254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.157890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.159412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1044/2097] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:35:44 2022] DELETE /identity/v3/projects/c57365923b844fe9b29d9e223ad49ce4 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:45.163496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:45.165094 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:45.178398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da12a1-00e9-4a18-8e79-904a1e4a2be3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:45.238047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:45.238669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.238946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.239219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] PATH_INFO: `/v3/users/5f09d9a29b0c41c48408fef8affc54bd` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.240335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f09d9a29b0c41c48408fef8affc54bd)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:45.242314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:45.288781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f09d9a29b0c41c48408fef8affc54bd'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.300681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f09d9a29b0c41c48408fef8affc54bd'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.301390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f09d9a29b0c41c48408fef8affc54bd'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.308653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f09d9a29b0c41c48408fef8affc54bd'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.309490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-bd00618d-a28f-4e56-bed1-fbaf8ba1a7e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f09d9a29b0c41c48408fef8affc54bd'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.322320 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1054/2098] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:35:45 2022] DELETE /identity/v3/users/5f09d9a29b0c41c48408fef8affc54bd => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:45.510091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:45.510091 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:45.524365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ee0159-949c-4a55-bd88-825a2b2e0186 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:45.591806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:45.592445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.592741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.593019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] PATH_INFO: `/v3/projects/7dc12b5da7a147b0a65eec8d04802cd2` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.594129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] RBAC: Authorizing `identity:delete_project(project_id=7dc12b5da7a147b0a65eec8d04802cd2)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:45.596074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:45.639805 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:45.648263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7dc12b5da7a147b0a65eec8d04802cd2', 'request_id': 'req-cccea25d-07de-4955-a135-82e5a162d01a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.654141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7dc12b5da7a147b0a65eec8d04802cd2', 'request_id': 'req-cccea25d-07de-4955-a135-82e5a162d01a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.660757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7dc12b5da7a147b0a65eec8d04802cd2', 'request_id': 'req-cccea25d-07de-4955-a135-82e5a162d01a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.673453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] The token cache is being invalidate because project 7dc12b5da7a147b0a65eec8d04802cd2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:35:45.673760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.674376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cccea25d-07de-4955-a135-82e5a162d01a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:45.675860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1045/2099] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:35:45 2022] DELETE /identity/v3/projects/7dc12b5da7a147b0a65eec8d04802cd2 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:45.681268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.681738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.681833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.704458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:45.705686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:45.765077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1055/2100] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:45.773234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.773595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.773956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.794883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:45.796088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:45.803991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1046/2101] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:35:45.808875 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:45.820037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d877d8-c778-4613-b807-8b12c9c311fc None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:45.827952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:45.827952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.827952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.827952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.828312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:35:45.828312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:45.829296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:45.844431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1056/2102] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:35:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:35:45.849674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:45.861242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f412ff-930e-449f-a771-f703b8361b6a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:45.866240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:45.866839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:45.867116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:45.867386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:45.868646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:45.870623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:45.916865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1047/2103] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:35:45 2022] POST /identity/v3/projects => generated 371 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:35:45.918568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:45.927278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:45.932142 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4795e25-9036-4e4b-8839-7b60666022f2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:46.005709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:46.006300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:46.006568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:46.006831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:46.007450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:46.009429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:46.030882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1057/2104] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:35:45 2022] POST /identity/v3/users => generated 286 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:35:46.035043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:46.046399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8fe78-85e8-4a40-b507-a280aed8f393 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:46.050955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:46.051543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:46.051815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:46.052081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:46.052733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:46.054360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:46.064820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1048/2105] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:46.068765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:46.078688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f4676f-6730-467b-82dc-aa1987d443aa admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:46.083297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a657aa-94ed-4c53-86dc-ae54706a2a8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:46.083889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a657aa-94ed-4c53-86dc-ae54706a2a8c admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:46.084159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a657aa-94ed-4c53-86dc-ae54706a2a8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:46.084423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a657aa-94ed-4c53-86dc-ae54706a2a8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:46.085090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a657aa-94ed-4c53-86dc-ae54706a2a8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:46.086687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a657aa-94ed-4c53-86dc-ae54706a2a8c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:46.100022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1058/2106] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:35:46.101338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:46.112081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f54ee-8844-467c-8f5d-afa3908d3738 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:46.116778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:46.117426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:46.118299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:46.118299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] PATH_INFO: `/v3/projects/b1601a3e4f41405d8e9001af6da86b46/users/60e6b5fcfd9d454bb10dd314bcdbedf6/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:46.134197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1601a3e4f41405d8e9001af6da86b46, user_id=60e6b5fcfd9d454bb10dd314bcdbedf6, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:46.136889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:46.150289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1049/2107] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:35:46 2022] PUT /identity/v3/projects/b1601a3e4f41405d8e9001af6da86b46/users/60e6b5fcfd9d454bb10dd314bcdbedf6/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:46.155413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c361b2e-720b-4d06-8092-42f919ec8323 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:46.155694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c361b2e-720b-4d06-8092-42f919ec8323 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:46.155960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c361b2e-720b-4d06-8092-42f919ec8323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:46.174466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6c361b2e-720b-4d06-8092-42f919ec8323 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:46.175687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6c361b2e-720b-4d06-8092-42f919ec8323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:46.233129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1059/2108] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:46.243727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:46.245562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:46.302045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97943fc7-c721-455e-9fd9-d36dbdfce58c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:46.306160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:46.306751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:46.307019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:46.307278 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:46.316384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:46.317838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:46.327969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1050/2109] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:50.166018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b851be2-efa6-4a73-b112-8d81ffcb6f79 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:50.168459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b851be2-efa6-4a73-b112-8d81ffcb6f79 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:50.168778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b851be2-efa6-4a73-b112-8d81ffcb6f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:50.212137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5b851be2-efa6-4a73-b112-8d81ffcb6f79 None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:50.217363 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5b851be2-efa6-4a73-b112-8d81ffcb6f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:50.352024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1060/2110] 10.222.0.23 () {58 vars in 1009 bytes} [Wed Oct 12 02:35:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:50.361880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:50.363824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:50.478325 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8776aba4-0ae2-4fd1-86f4-296eb1eef115 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:50.484360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:50.485384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:50.485669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:50.485968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:50.494428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:50.497545 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:50.499610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:50.517991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1051/2111] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:35:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 12 02:35:51.843114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0890b0-1907-472b-b799-372991462c84 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:51.844113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0890b0-1907-472b-b799-372991462c84 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:51.844607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0890b0-1907-472b-b799-372991462c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:51.888125 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-cc0890b0-1907-472b-b799-372991462c84 None None] MFA Rules not processed for user `60e6b5fcfd9d454bb10dd314bcdbedf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:51.889608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cc0890b0-1907-472b-b799-372991462c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:51.935815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:52.049747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1061/2112] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:52.050111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:52.062295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e379e93-fa92-4a39-9296-cbc29a0d190e service swift] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:52.067530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:52.068749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:52.068749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:52.068978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:52.071759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:52.073792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:52.124432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1052/2113] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:35:52 2022] POST /identity/v3/projects => generated 371 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:35:52.128483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0890b0-1907-472b-b799-372991462c84 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:52.156765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0890b0-1907-472b-b799-372991462c84 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:52.270305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:52.270925 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:52.271197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:52.271465 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:52.272107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:52.279684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:52.316517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1062/2114] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:35:52 2022] POST /identity/v3/users => generated 292 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:35:52.317711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:52.329473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43954ff9-a478-4245-93ab-fdbedfc97f6c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:52.334481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:52.335097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:52.335372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:52.335651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:52.336342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:52.338387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:52.351495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1053/2115] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:52.355843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:52.367310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2eb58-d70f-48cf-ae92-99c17441e50d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:52.378232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:52.379214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:52.379634 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:52.379999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] PATH_INFO: `/v3/projects/e17aa6bdc3204e02a0ffae37efac63b4/users/3ea8132514d646f8865347accfc23dda/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:52.409073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=e17aa6bdc3204e02a0ffae37efac63b4, user_id=3ea8132514d646f8865347accfc23dda, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:52.412034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:52.432582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1063/2116] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:35:52 2022] PUT /identity/v3/projects/e17aa6bdc3204e02a0ffae37efac63b4/users/3ea8132514d646f8865347accfc23dda/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:35:52.435763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:52.454273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0e3444-a98f-46e9-a0db-b09d6411a915 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:52.540782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:52.541419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:52.541693 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:52.541963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:52.542608 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:52.547344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:52.562617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1054/2117] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:35:52 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:35:52.567031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:52.579730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7ecc3d-39c2-4c3e-a946-7e002e0157fd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:52.590919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07161b8-dbc3-44d6-b845-1ad4edc71001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:52.591991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07161b8-dbc3-44d6-b845-1ad4edc71001 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:52.592577 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07161b8-dbc3-44d6-b845-1ad4edc71001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:52.595541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07161b8-dbc3-44d6-b845-1ad4edc71001 admin admin] PATH_INFO: `/v3/projects/e17aa6bdc3204e02a0ffae37efac63b4/users/3ea8132514d646f8865347accfc23dda/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:52.599006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07161b8-dbc3-44d6-b845-1ad4edc71001 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e17aa6bdc3204e02a0ffae37efac63b4, user_id=3ea8132514d646f8865347accfc23dda, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:52.603275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07161b8-dbc3-44d6-b845-1ad4edc71001 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:52.635424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1064/2118] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:35:52 2022] PUT /identity/v3/projects/e17aa6bdc3204e02a0ffae37efac63b4/users/3ea8132514d646f8865347accfc23dda/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:55.725063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:55.727501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:55.779297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49b06b-4148-49f6-ba32-f99f5f27d4c0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:55.978268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:55.979116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:55.979402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:55.979676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] PATH_INFO: `/v3/users/28b883e86f6e44688767b09cd661c7b1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:55.980969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=28b883e86f6e44688767b09cd661c7b1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:55.984035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:56.093783 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28b883e86f6e44688767b09cd661c7b1'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:56.132532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:INSERT=1,keystone:DELETE=8 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:56.139980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28b883e86f6e44688767b09cd661c7b1'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:56.141472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28b883e86f6e44688767b09cd661c7b1'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:56.155196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28b883e86f6e44688767b09cd661c7b1'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:56.155929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28b883e86f6e44688767b09cd661c7b1'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:56.178236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1055/2119] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:35:55 2022] DELETE /identity/v3/users/28b883e86f6e44688767b09cd661c7b1 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:56.182583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:56.183103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:56.183200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:56.244587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:56.246201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:56.394002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1065/2120] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:56.406657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:56.408463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:56.510508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a70fe5-cfba-41a9-8ccc-e20b531a21fb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:56.520632 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:56.522431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:56.525794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:56.526108 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:56.551164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:56.553373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:56.594158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1056/2121] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:35:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:35:57.639612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:57.643226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:57.684248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97877c7-28be-4f64-af0f-470c08f78ed9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:57.723497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:57.724208 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:57.724484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:57.724865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] PATH_INFO: `/v3/projects/2e7dfa954e634c46b53e71b39dde73ac` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:57.728566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e7dfa954e634c46b53e71b39dde73ac)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:57.730699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:57.745241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:57.881745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e7dfa954e634c46b53e71b39dde73ac', 'request_id': 'req-2acb6e18-1e40-41b9-a7c6-c8a005176b07'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:57.906904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e7dfa954e634c46b53e71b39dde73ac', 'request_id': 'req-2acb6e18-1e40-41b9-a7c6-c8a005176b07'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:57.916557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e7dfa954e634c46b53e71b39dde73ac', 'request_id': 'req-2acb6e18-1e40-41b9-a7c6-c8a005176b07'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:57.948587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] The token cache is being invalidate because project 2e7dfa954e634c46b53e71b39dde73ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:35:57.948921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:57.949631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:57.951836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1066/2122] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:35:57 2022] DELETE /identity/v3/projects/2e7dfa954e634c46b53e71b39dde73ac => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:57.955860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:57.985081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a22733-2c27-4ab6-b0b4-4866bf9e1f28 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:58.077243 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:58.078293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:58.078740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:58.079204 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] PATH_INFO: `/v3/users/b934dd26d5a042b2b42d1e415b003fbc` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:58.081604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b934dd26d5a042b2b42d1e415b003fbc)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:58.085518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:58.179201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b934dd26d5a042b2b42d1e415b003fbc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:58.217125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b934dd26d5a042b2b42d1e415b003fbc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:58.217993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b934dd26d5a042b2b42d1e415b003fbc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:58.232110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b934dd26d5a042b2b42d1e415b003fbc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:58.232905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9e18a457-9f98-4215-98fb-f3d0e949862b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b934dd26d5a042b2b42d1e415b003fbc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:58.259631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:58.273362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1057/2123] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:35:57 2022] DELETE /identity/v3/users/b934dd26d5a042b2b42d1e415b003fbc => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:59.276074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:59.279258 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:59.321177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acb6e18-1e40-41b9-a7c6-c8a005176b07 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:59.444399 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:59.446177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:59.446775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:59.447207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] PATH_INFO: `/v3/projects/5842ddf8c54546b18ae0b06eb2e78b40` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:59.449228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5842ddf8c54546b18ae0b06eb2e78b40)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:59.453328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:59.568392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5842ddf8c54546b18ae0b06eb2e78b40', 'request_id': 'req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:59.584267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5842ddf8c54546b18ae0b06eb2e78b40', 'request_id': 'req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:59.595757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5842ddf8c54546b18ae0b06eb2e78b40', 'request_id': 'req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:59.623696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] The token cache is being invalidate because project 5842ddf8c54546b18ae0b06eb2e78b40 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:35:59.625620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:59.627200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-0b030fc3-c1c2-4876-b2fe-b1f2289477d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:35:59.632087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1067/2124] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:35:59 2022] DELETE /identity/v3/projects/5842ddf8c54546b18ae0b06eb2e78b40 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:35:59.635768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:59.636291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:59.636504 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:59.678720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:59.686350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:59.807904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1058/2125] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:59.809508 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:59.809798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:59.810068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:59.850052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:35:59.851329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:35:59.864013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1068/2126] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:35:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:35:59.865331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:59.875625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19ce016-4cc8-4f5d-9ba3-fc2845b3964c None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:59.880390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:59.881035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:59.881307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:59.881576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:59.882226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:35:59.882524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:59.884195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:59.905539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1059/2127] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:35:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:35:59.906739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:59.921129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a7b57-1b12-450e-aaf6-cf0e1b0fceff None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:35:59.926625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:35:59.927259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:35:59.927534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:35:59.927802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:35:59.928439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:35:59.930434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:35:59.948993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:35:59.972169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1069/2128] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:35:59 2022] POST /identity/v3/projects => generated 369 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:35:59.977299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:35:59.991696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6236b1e-3cf2-4f8e-8958-ce25b4bc8eb1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.078747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.079443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.079671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.079883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.081071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.089404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.120400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1060/2129] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:35:59 2022] POST /identity/v3/users => generated 285 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:00.124560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.137805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1e19ef-8ec9-4d2f-8da4-d659c166b46e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.144813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.145595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.145973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.146462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.147088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.150670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.168789 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1070/2130] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:00 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:00.170421 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.184041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c6bde3-e54c-4fc4-bce6-b4d978775312 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.191547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc0dc5f-4e79-4c63-8559-e99a5c4195f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.192476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0dc5f-4e79-4c63-8559-e99a5c4195f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.192599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0dc5f-4e79-4c63-8559-e99a5c4195f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.192910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0dc5f-4e79-4c63-8559-e99a5c4195f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.193624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc0dc5f-4e79-4c63-8559-e99a5c4195f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.195945 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc0dc5f-4e79-4c63-8559-e99a5c4195f0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.225772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1061/2131] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:00 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:00.226978 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.238569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8bfdb-f9aa-40e5-94cb-7a00ac550e66 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.244676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356dee56-0b8c-45ea-9448-581d37caa4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.245506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356dee56-0b8c-45ea-9448-581d37caa4bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.246055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356dee56-0b8c-45ea-9448-581d37caa4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.246973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356dee56-0b8c-45ea-9448-581d37caa4bc admin admin] PATH_INFO: `/v3/projects/c956744f6bd34a6994fe3e9a93a1b5ca/users/08e1054aa04248c3b8563641c5e5b2ad/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.280593 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356dee56-0b8c-45ea-9448-581d37caa4bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=c956744f6bd34a6994fe3e9a93a1b5ca, user_id=08e1054aa04248c3b8563641c5e5b2ad, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.289900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356dee56-0b8c-45ea-9448-581d37caa4bc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.310030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1071/2132] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:00 2022] PUT /identity/v3/projects/c956744f6bd34a6994fe3e9a93a1b5ca/users/08e1054aa04248c3b8563641c5e5b2ad/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:00.321161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.321445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.321719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.363232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] MFA Rules not processed for user `08e1054aa04248c3b8563641c5e5b2ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:00.365055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:00.430974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:00.435979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.437002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.437941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.444887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1062/2133] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:00 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:00.447477 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.478914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] MFA Rules not processed for user `3ea8132514d646f8865347accfc23dda`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:00.481301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:00.550791 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d3c96-779a-4c40-bae9-3f9463599034 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.556200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.556846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.557125 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.557396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.558035 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.560321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.602082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1063/2134] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:00 2022] POST /identity/v3/projects => generated 367 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:00.605574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.625380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1072/2135] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:00 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:00.640961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.643442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:00.710184 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce11c0cc-c2d5-481d-bf40-d8f5a109f6e5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.715412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.716059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.716336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.716604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.717269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.719293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.747553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1064/2136] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:00 2022] POST /identity/v3/users => generated 290 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:00.751367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.752933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e411f000-53dd-4931-873c-5e19d1e2043f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.764185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e603ec-1adc-444b-8f38-dfb502c7e328 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.769472 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.770790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.770790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.770790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.771527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.773254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.788307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1065/2137] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:00 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:00.789819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.800020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69e1dd2-56eb-46ba-8117-01b23126695e admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.805685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.806476 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.806772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.807061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] PATH_INFO: `/v3/projects/c4555e3438da44e09ebb9233502d8883/users/95cd4862df524657a99c783e204888e4/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.826242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4555e3438da44e09ebb9233502d8883, user_id=95cd4862df524657a99c783e204888e4, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.831821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.835987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:00.852077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1066/2138] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:00 2022] PUT /identity/v3/projects/c4555e3438da44e09ebb9233502d8883/users/95cd4862df524657a99c783e204888e4/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:00.869475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.883687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.884309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.884586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.885014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.889422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:00.919836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:00.951034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a227d12f-4bc0-4ffd-8f90-bdf5664c770f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.955902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.956384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.957568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.957568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.957568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.958202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:00.960023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.961262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674ba060-0f80-4cee-9ede-2c98e5cb6c5d service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:00.971335 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1067/2139] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:00 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:00.971692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1073/2140] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:00 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:36:00.975530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:00.987946 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8217bc-bd04-4fa0-aa6c-8f4cab151fa2 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:00.993903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:00.994838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:00.995126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:00.995377 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] PATH_INFO: `/v3/projects/c4555e3438da44e09ebb9233502d8883/users/95cd4862df524657a99c783e204888e4/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:00.997410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4555e3438da44e09ebb9233502d8883, user_id=95cd4862df524657a99c783e204888e4, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:01.000072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:01.015119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1068/2141] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:00 2022] PUT /identity/v3/projects/c4555e3438da44e09ebb9233502d8883/users/95cd4862df524657a99c783e204888e4/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:01.026579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04976aab-03fa-4a34-bef3-7ee3d5c76182 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:01.026890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04976aab-03fa-4a34-bef3-7ee3d5c76182 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:01.027031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04976aab-03fa-4a34-bef3-7ee3d5c76182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:01.067983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-04976aab-03fa-4a34-bef3-7ee3d5c76182 None None] MFA Rules not processed for user `95cd4862df524657a99c783e204888e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:01.069321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-04976aab-03fa-4a34-bef3-7ee3d5c76182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:01.178745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1074/2142] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:01 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:01.196201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:01.198109 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:01.281276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101c0c2d-5e6b-4d6b-99d5-0f40621abaf8 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:01.285564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:01.286271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:01.286614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:01.286954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:01.298068 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:01.299624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:01.319410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1069/2143] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:01 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:02.166040 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1263065-1221-4fe2-b1a3-e75499232f0d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:02.166480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1263065-1221-4fe2-b1a3-e75499232f0d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:02.166689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1263065-1221-4fe2-b1a3-e75499232f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:02.186660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f1263065-1221-4fe2-b1a3-e75499232f0d None None] MFA Rules not processed for user `60e6b5fcfd9d454bb10dd314bcdbedf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:02.187944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f1263065-1221-4fe2-b1a3-e75499232f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:02.251188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1075/2144] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:02.921575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:03.009189 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d42a36-ca2e-4a46-82c4-df69365a7a41 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:03.014228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:03.014777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:03.015053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:03.015322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] PATH_INFO: `/v3/users/08e1054aa04248c3b8563641c5e5b2ad` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:03.016442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08e1054aa04248c3b8563641c5e5b2ad)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:03.018520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:03.068850 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08e1054aa04248c3b8563641c5e5b2ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.080514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:INSERT=1,keystone:DELETE=9 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:03.082019 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08e1054aa04248c3b8563641c5e5b2ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.083008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08e1054aa04248c3b8563641c5e5b2ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.092496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08e1054aa04248c3b8563641c5e5b2ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.093437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08e1054aa04248c3b8563641c5e5b2ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.117428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1070/2145] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:02 2022] DELETE /identity/v3/users/08e1054aa04248c3b8563641c5e5b2ad => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:03.117712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:03.117993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:03.118439 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:03.142293 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:03.149143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:03.156151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:03.232479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1076/2146] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:03.242305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:03.243735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:03.318363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b93c3e-c17b-43c8-a868-eefa3d3a9e83 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:03.322706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:03.323552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:03.323906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:03.324230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:03.334361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:03.336481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:03.347038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1071/2147] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:03.659954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:03.664395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:03.677426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a0b55b-cd6d-493c-9a2d-efb1e94a266b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:03.686852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:03.687238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:03.687513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:03.688030 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] PATH_INFO: `/v3/projects/c956744f6bd34a6994fe3e9a93a1b5ca` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:03.689730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c956744f6bd34a6994fe3e9a93a1b5ca)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:03.691622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:03.753103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c956744f6bd34a6994fe3e9a93a1b5ca', 'request_id': 'req-3a938a99-b483-4399-80dd-66f01b4a2978'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.759549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c956744f6bd34a6994fe3e9a93a1b5ca', 'request_id': 'req-3a938a99-b483-4399-80dd-66f01b4a2978'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.767860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c956744f6bd34a6994fe3e9a93a1b5ca', 'request_id': 'req-3a938a99-b483-4399-80dd-66f01b4a2978'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.781288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] The token cache is being invalidate because project c956744f6bd34a6994fe3e9a93a1b5ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:03.781542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.782250 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.784027 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1077/2148] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:03 2022] DELETE /identity/v3/projects/c956744f6bd34a6994fe3e9a93a1b5ca => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:03.788028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:03.799216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f5be50-3a27-47ae-a89a-22cb11e368f3 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:03.877740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:03.880732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:03.880732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:03.880732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] PATH_INFO: `/v3/users/95cd4862df524657a99c783e204888e4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:03.880732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95cd4862df524657a99c783e204888e4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:03.882983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:03.950555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95cd4862df524657a99c783e204888e4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.967858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95cd4862df524657a99c783e204888e4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.968837 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95cd4862df524657a99c783e204888e4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.978200 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95cd4862df524657a99c783e204888e4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.978950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-32db0475-7535-442a-96ab-1a3baf81a115 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95cd4862df524657a99c783e204888e4'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:03.997698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1072/2149] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:03 2022] DELETE /identity/v3/users/95cd4862df524657a99c783e204888e4 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:04.304885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:04.306730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:04.320681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a938a99-b483-4399-80dd-66f01b4a2978 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:04.390168 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:04.390799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.391068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.391355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] PATH_INFO: `/v3/projects/c4555e3438da44e09ebb9233502d8883` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.392457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4555e3438da44e09ebb9233502d8883)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:04.395214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:04.440302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:04.452663 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4555e3438da44e09ebb9233502d8883', 'request_id': 'req-11afa436-0160-4241-a1c6-7b7c93413fc9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:04.458893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4555e3438da44e09ebb9233502d8883', 'request_id': 'req-11afa436-0160-4241-a1c6-7b7c93413fc9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:04.470666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4555e3438da44e09ebb9233502d8883', 'request_id': 'req-11afa436-0160-4241-a1c6-7b7c93413fc9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:04.484229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] The token cache is being invalidate because project c4555e3438da44e09ebb9233502d8883 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:04.484378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:04.484999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-11afa436-0160-4241-a1c6-7b7c93413fc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:04.486502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1078/2150] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:04 2022] DELETE /identity/v3/projects/c4555e3438da44e09ebb9233502d8883 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:04.491435 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.491789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.492063 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.499552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:04.514172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:04.515599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:04.605470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1073/2151] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:04.613383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.613662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.613930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.636063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:04.637883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:04.646581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1079/2152] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:04.651351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:04.702943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43865d0e-92b3-46ae-a01c-0fcd191a1868 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:04.710012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:04.711327 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.711668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.712408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.714104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:04.714717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:04.716930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:04.733746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1074/2153] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:36:04.744895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:04.780914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12f9ae-62d0-4798-a5cb-db21b0de1d70 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:04.787855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:04.788498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.789238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.789659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.790310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:04.792838 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:04.832552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1080/2154] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:04 2022] POST /identity/v3/projects => generated 387 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:04.833763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:04.845696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db25af6-2c4f-4ab1-9b0a-c376ce53223a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:04.911671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:04.912394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.912672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.912969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.913599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:04.915555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:04.943475 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1075/2155] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:04 2022] POST /identity/v3/users => generated 294 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:04.948898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:04.959918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfdbf10-7d63-4326-9d40-09b036bec391 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:04.965473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:04.966075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:04.966348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:04.966618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:04.967261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:04.968907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:04.983550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1081/2156] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:04.985537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:04.997186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15c42f7-efed-4f43-ae4e-afce6838e3bd admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.002219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0bfc46-0f04-494a-b149-c2e38d17bfad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.002824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0bfc46-0f04-494a-b149-c2e38d17bfad admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.003098 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0bfc46-0f04-494a-b149-c2e38d17bfad admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.003366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0bfc46-0f04-494a-b149-c2e38d17bfad admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.004008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0bfc46-0f04-494a-b149-c2e38d17bfad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.005675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0bfc46-0f04-494a-b149-c2e38d17bfad admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.017707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1076/2157] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:05.021782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.032392 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35caac7f-ed3f-4cf1-9121-f06b29003310 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.037281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.037966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.038243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.038512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] PATH_INFO: `/v3/projects/aaf7744423654345a8999504bc943166/users/72f445967e56408ca919c46e54c50977/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.059088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaf7744423654345a8999504bc943166, user_id=72f445967e56408ca919c46e54c50977, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.061953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.078249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1082/2158] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:05 2022] PUT /identity/v3/projects/aaf7744423654345a8999504bc943166/users/72f445967e56408ca919c46e54c50977/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:05.089706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.089984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.090256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.140650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] MFA Rules not processed for user `72f445967e56408ca919c46e54c50977`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:05.142051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:05.240100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:05.265379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1077/2159] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:05.277387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.349491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e4c575-4946-44c0-813d-d3a2311e3857 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.354855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.355482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.355758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.356025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.359202 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.360426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.394861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1083/2160] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:05 2022] POST /identity/v3/projects => generated 387 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:05.399694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.410982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae83eeba-7861-42be-9058-0b24fbb95201 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.479143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.479913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.480260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.480536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.481206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.483403 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.511862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1078/2161] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:05 2022] POST /identity/v3/users => generated 294 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:05.513275 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.527879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c86a2cd-232a-481f-9c45-4d395df5ba79 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.534370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.535002 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.535278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.535543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.536182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.537856 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.551467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1084/2162] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:05.556397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.567502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3e1bac-edef-41ba-821b-c408cd19f4a8 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.575332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad579a-dd20-46f2-9e87-a1309799a165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.575959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad579a-dd20-46f2-9e87-a1309799a165 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.576277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad579a-dd20-46f2-9e87-a1309799a165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.576548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad579a-dd20-46f2-9e87-a1309799a165 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.577367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ad579a-dd20-46f2-9e87-a1309799a165 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.579194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ad579a-dd20-46f2-9e87-a1309799a165 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.594917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1079/2163] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:05.599466 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.610681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44fc7a1-5198-464a-adf7-feb6f495a8d6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.615510 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.616176 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.616445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.617352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] PATH_INFO: `/v3/projects/2481b79a33cf467b98c7a595b7a4fc49/users/33461225687744af8aac02d1435c692f/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.633473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2481b79a33cf467b98c7a595b7a4fc49, user_id=33461225687744af8aac02d1435c692f, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.636011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.657705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1085/2164] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:05 2022] PUT /identity/v3/projects/2481b79a33cf467b98c7a595b7a4fc49/users/33461225687744af8aac02d1435c692f/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:05.669092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.669376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.669739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.699226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] MFA Rules not processed for user `33461225687744af8aac02d1435c692f`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:05.700493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:05.775337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1080/2165] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:05.780000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.793242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:05.847284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7be1a-a9c2-470f-8c49-38422621ca3a admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.852501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.853134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.853409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.853677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.854324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.856254 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.887114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1086/2166] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:05 2022] POST /identity/v3/projects => generated 385 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:05.891166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:05.903291 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103cc74-3e60-46e8-86eb-57e615466c48 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:05.964870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:05.965485 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:05.965763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:05.966034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:05.966674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:05.968616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:05.981533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:05.995045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1081/2167] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:05 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:05.999186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.010573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaece4a-3e0b-4be3-bba0-021454f55f9a admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:06.015153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:06.015783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.016052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.016315 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.017004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:06.018591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:06.031731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1087/2168] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:06.035949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.045980 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e9bbd-57f4-4761-b5a2-98e5fc105ee4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:06.050496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:06.051132 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.051405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.051670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] PATH_INFO: `/v3/projects/48d7d3fc40a042a8958dfbda71b4a73e/users/c2c331eca0ba41cca00711b69407e374/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.067718 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48d7d3fc40a042a8958dfbda71b4a73e, user_id=c2c331eca0ba41cca00711b69407e374, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:06.070952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:06.083396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1082/2169] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:06 2022] PUT /identity/v3/projects/48d7d3fc40a042a8958dfbda71b4a73e/users/c2c331eca0ba41cca00711b69407e374/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:06.088497 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.100057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed13b4-caa5-4e54-a4ba-9b7b60a1979c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:06.158962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877c2f30-8c8b-4586-938a-1559a0a51bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:06.159490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877c2f30-8c8b-4586-938a-1559a0a51bd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.159767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877c2f30-8c8b-4586-938a-1559a0a51bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.160033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877c2f30-8c8b-4586-938a-1559a0a51bd1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.160668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877c2f30-8c8b-4586-938a-1559a0a51bd1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:06.162336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877c2f30-8c8b-4586-938a-1559a0a51bd1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:06.172636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1088/2170] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:06 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:06.177016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.187179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad33cfd7-59a6-4fb0-8ef0-04e8f69da062 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:06.192372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:06.193064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.193339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.193612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] PATH_INFO: `/v3/projects/48d7d3fc40a042a8958dfbda71b4a73e/users/c2c331eca0ba41cca00711b69407e374/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.196064 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48d7d3fc40a042a8958dfbda71b4a73e, user_id=c2c331eca0ba41cca00711b69407e374, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:06.198754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:06.209955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1083/2171] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:06 2022] PUT /identity/v3/projects/48d7d3fc40a042a8958dfbda71b4a73e/users/c2c331eca0ba41cca00711b69407e374/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:06.220968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c777307-04e5-44d7-9843-2b99b9f05d85 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.221251 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c777307-04e5-44d7-9843-2b99b9f05d85 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.221520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c777307-04e5-44d7-9843-2b99b9f05d85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.249951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-7c777307-04e5-44d7-9843-2b99b9f05d85 None None] MFA Rules not processed for user `c2c331eca0ba41cca00711b69407e374`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:06.251213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7c777307-04e5-44d7-9843-2b99b9f05d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:06.325324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1089/2172] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:06 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:06.336668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.338426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:06.408244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382ae4c-fd3a-4013-8166-e916c75a8e50 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:06.411955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:06.412498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.412806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.413076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.422552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:06.424000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:06.436663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1084/2173] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:06 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:06.840874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:06.841351 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:06.841630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:06.857387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:06.867666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] MFA Rules not processed for user `f8d2a8079f40422c8bdb9b73b391d0c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:06.869092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:06.930224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1090/2174] 10.222.0.23 () {58 vars in 967 bytes} [Wed Oct 12 02:36:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:06.943909 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.945535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:06.982828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:06.995204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53208a8-24d2-42d4-aa9c-e70bec89b10b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:06.999562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:07.000287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:07.000628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:07.001321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:07.042538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461317f-b4dc-4250-a066-b64eb6481227 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:07.047646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c57115b9feff40629f5081e32ff4bad3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:07.048279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:07.048553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:07.048856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:07.058579 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:07.059705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:07.060487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:07.061328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:07.069673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1085/2175] 10.222.0.23 () {60 vars in 1384 bytes} [Wed Oct 12 02:36:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 12 02:36:07.075692 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1091/2176] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:06 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:11.222276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:11.232722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c900ed1-6381-4c6e-85f9-ae44b81ebad5 service swift] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:11.236310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d72bca-a18a-4bed-b4ec-9aa5a3d91d9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:11.236936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d72bca-a18a-4bed-b4ec-9aa5a3d91d9d service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:11.237214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d72bca-a18a-4bed-b4ec-9aa5a3d91d9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:11.237482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d72bca-a18a-4bed-b4ec-9aa5a3d91d9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:11.286213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:11.296871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d72bca-a18a-4bed-b4ec-9aa5a3d91d9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:11.298339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d72bca-a18a-4bed-b4ec-9aa5a3d91d9d service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:11.308286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1086/2177] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:11 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:13.416976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:13.478912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3145bf3f-aca4-49cc-a6bf-324a3022571e service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:13.484198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:13.484847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:13.485124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:13.485486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] PATH_INFO: `/v3/users/72f445967e56408ca919c46e54c50977` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:13.486496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72f445967e56408ca919c46e54c50977)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:13.488403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:13.534280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72f445967e56408ca919c46e54c50977'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:13.538632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:13.548754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72f445967e56408ca919c46e54c50977'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:13.549440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72f445967e56408ca919c46e54c50977'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:13.556411 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72f445967e56408ca919c46e54c50977'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:13.557173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72f445967e56408ca919c46e54c50977'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:13.570844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1092/2178] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:13 2022] DELETE /identity/v3/users/72f445967e56408ca919c46e54c50977 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:13.575378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:13.575941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:13.575941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:13.595210 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:13.596723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:13.657457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1087/2179] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:13.667171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:13.669166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:13.726806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a74ddc7-f850-44cc-b445-984e5b03d664 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:13.730967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:13.731820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:13.732099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:13.732383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:13.741223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:13.742641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:13.752442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1093/2180] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:14.025139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:14.026547 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:14.038995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fc161-407b-4922-a0b5-543fd5ebf499 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:14.044932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:14.045563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:14.045914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:14.046157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] PATH_INFO: `/v3/projects/aaf7744423654345a8999504bc943166` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:14.047290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaf7744423654345a8999504bc943166)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:14.049253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:14.096390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaf7744423654345a8999504bc943166', 'request_id': 'req-d30ed5d3-953f-4f6e-8605-b27443e19990'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.102323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaf7744423654345a8999504bc943166', 'request_id': 'req-d30ed5d3-953f-4f6e-8605-b27443e19990'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.109550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaf7744423654345a8999504bc943166', 'request_id': 'req-d30ed5d3-953f-4f6e-8605-b27443e19990'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.123784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] The token cache is being invalidate because project aaf7744423654345a8999504bc943166 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:14.124066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.124669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.126164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1088/2181] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:14 2022] DELETE /identity/v3/projects/aaf7744423654345a8999504bc943166 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:14.130191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:14.139702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae4a66-e132-483e-8e8f-812ba57b00a9 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:14.199206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:14.199847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:14.200122 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:14.200390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] PATH_INFO: `/v3/users/33461225687744af8aac02d1435c692f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:14.201554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] RBAC: Authorizing `identity:delete_user(user_id=33461225687744af8aac02d1435c692f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:14.203471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:14.250951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33461225687744af8aac02d1435c692f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.263622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33461225687744af8aac02d1435c692f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.264454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33461225687744af8aac02d1435c692f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.271402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33461225687744af8aac02d1435c692f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.272079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33461225687744af8aac02d1435c692f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.274280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:14.286641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1094/2182] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:14 2022] DELETE /identity/v3/users/33461225687744af8aac02d1435c692f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:14.551813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:14.553334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:14.571663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30ed5d3-953f-4f6e-8605-b27443e19990 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:14.627558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:14.628181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:14.628441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:14.628723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] PATH_INFO: `/v3/projects/2481b79a33cf467b98c7a595b7a4fc49` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:14.629845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] RBAC: Authorizing `identity:delete_project(project_id=2481b79a33cf467b98c7a595b7a4fc49)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:14.631723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:14.647703 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:14.678033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2481b79a33cf467b98c7a595b7a4fc49', 'request_id': 'req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.684061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2481b79a33cf467b98c7a595b7a4fc49', 'request_id': 'req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.690246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2481b79a33cf467b98c7a595b7a4fc49', 'request_id': 'req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.703029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] The token cache is being invalidate because project 2481b79a33cf467b98c7a595b7a4fc49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:14.703507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.704166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.705688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1089/2183] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:14 2022] DELETE /identity/v3/projects/2481b79a33cf467b98c7a595b7a4fc49 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:14.709790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:14.711301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:14.723648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01faad8d-d2e8-418b-ae1c-c74fa8236500 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:14.779814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:14.780442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:14.780743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:14.781015 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] PATH_INFO: `/v3/users/c2c331eca0ba41cca00711b69407e374` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:14.782131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2c331eca0ba41cca00711b69407e374)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:14.784044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:14.828541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2c331eca0ba41cca00711b69407e374'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.840551 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2c331eca0ba41cca00711b69407e374'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.841241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2c331eca0ba41cca00711b69407e374'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.848128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2c331eca0ba41cca00711b69407e374'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.848813 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4a540bee-3a45-4c13-9468-84ee174722bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2c331eca0ba41cca00711b69407e374'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:14.861162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1095/2184] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:14 2022] DELETE /identity/v3/users/c2c331eca0ba41cca00711b69407e374 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:15.127050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.128494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:15.141137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41df35ad-aff0-4c4f-9b9a-5363c4faedfb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.199129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.199751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.200022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.200287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] PATH_INFO: `/v3/projects/48d7d3fc40a042a8958dfbda71b4a73e` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.201431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48d7d3fc40a042a8958dfbda71b4a73e)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.203323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.249985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48d7d3fc40a042a8958dfbda71b4a73e', 'request_id': 'req-cec18367-cae3-4bd6-ba6b-17c929086da5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:15.255681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48d7d3fc40a042a8958dfbda71b4a73e', 'request_id': 'req-cec18367-cae3-4bd6-ba6b-17c929086da5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:15.262231 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48d7d3fc40a042a8958dfbda71b4a73e', 'request_id': 'req-cec18367-cae3-4bd6-ba6b-17c929086da5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:15.276124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] The token cache is being invalidate because project 48d7d3fc40a042a8958dfbda71b4a73e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:15.276416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:15.277991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-cec18367-cae3-4bd6-ba6b-17c929086da5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:15.279496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1090/2185] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:15 2022] DELETE /identity/v3/projects/48d7d3fc40a042a8958dfbda71b4a73e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:15.285391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.285670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.285940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.306033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:15.307213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:15.364295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1096/2186] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:15.372534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.372810 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.373079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.392163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:15.393368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:15.401010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1091/2187] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:15.405539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.415189 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fb7631-2351-482c-a6b3-b1d4f6a7b8d6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.420672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.421342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.421621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.421887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.422535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:15.422840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.424430 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.431349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:15.436823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1097/2188] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:36:15.441202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.450407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90614032-6861-46f9-8e12-5ab708a5f7b8 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.455255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.455859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.456136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.456406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.457078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.458984 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.487710 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1092/2189] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/projects => generated 387 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:15.491819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.501591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05610e6-0596-4607-99cb-9a7fb32dd195 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.555499 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.556084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.556355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.556618 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.557265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.559177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.579396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1098/2190] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/users => generated 294 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:15.583914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.591265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:15.595431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eff873-9f4d-4cbb-a283-b28ba680b338 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.601922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.602541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.603033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.603361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.604160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.606668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.616500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1093/2191] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:15.620355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.630238 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c8c1b-de7a-4294-971e-f55f989b2e01 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.634724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31337941-f02f-4aba-828c-a721a9b7d3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.635309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31337941-f02f-4aba-828c-a721a9b7d3b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.635583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31337941-f02f-4aba-828c-a721a9b7d3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.635851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31337941-f02f-4aba-828c-a721a9b7d3b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.636479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31337941-f02f-4aba-828c-a721a9b7d3b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.638093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31337941-f02f-4aba-828c-a721a9b7d3b2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.648454 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1099/2192] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:15.652993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.663158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3b2917-1783-483a-941d-f62ef59bb275 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.668366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.669038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.669314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.669583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] PATH_INFO: `/v3/projects/38377b9a9a6a4c75be150b4089bf7a73/users/f6d388da6e304502ab21382ec735001a/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.684293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38377b9a9a6a4c75be150b4089bf7a73, user_id=f6d388da6e304502ab21382ec735001a, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.686981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.700402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1094/2193] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:15 2022] PUT /identity/v3/projects/38377b9a9a6a4c75be150b4089bf7a73/users/f6d388da6e304502ab21382ec735001a/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:15.711568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.712029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.712150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.740112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] MFA Rules not processed for user `f6d388da6e304502ab21382ec735001a`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:15.741340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:15.804332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1100/2194] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:15.808796 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.870339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7803fa-524a-4d41-b48e-e3556cbdaec2 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.875097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.875705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.875981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.876250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.876902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.878836 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.910886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1095/2195] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:15.912069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:15.922781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a60cdc0-55d9-41d8-a427-511124211c1b None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:15.977268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:15.977866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:15.978148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:15.978397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:15.979012 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:15.980988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:15.991988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:16.005032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1101/2196] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:15 2022] POST /identity/v3/users => generated 299 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:16.009526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:16.021037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44555f6f-4dd9-4c87-b93e-efa421235160 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:16.026786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:16.027432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:16.027753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:16.028052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:16.028831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:16.031161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:16.041811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1096/2197] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:16.046052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:16.055830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f97b2e0-ba91-4e98-8e9d-a7561c945c8c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:16.061152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:16.061797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:16.062066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:16.062328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] PATH_INFO: `/v3/projects/20a8f3dcbed2483a9503809231ceb898/users/c0781f1357314dd29217c8a1c21a4b5a/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:16.078704 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20a8f3dcbed2483a9503809231ceb898, user_id=c0781f1357314dd29217c8a1c21a4b5a, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:16.081442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:16.093585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1102/2198] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:16 2022] PUT /identity/v3/projects/20a8f3dcbed2483a9503809231ceb898/users/c0781f1357314dd29217c8a1c21a4b5a/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:16.097482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:16.107806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a739a-cb9d-450b-bb5d-7bad3f2b1d4d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:16.162784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870468f3-e68a-4c67-816c-0baa41886e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:16.163309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870468f3-e68a-4c67-816c-0baa41886e17 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:16.163580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870468f3-e68a-4c67-816c-0baa41886e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:16.163844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870468f3-e68a-4c67-816c-0baa41886e17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:16.164465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870468f3-e68a-4c67-816c-0baa41886e17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:16.166099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870468f3-e68a-4c67-816c-0baa41886e17 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:16.175749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1097/2199] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:16.179957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:16.190455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc6855c-54aa-486d-956f-6d827a313f04 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:16.194973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:16.195621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:16.195894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:16.196161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] PATH_INFO: `/v3/projects/20a8f3dcbed2483a9503809231ceb898/users/c0781f1357314dd29217c8a1c21a4b5a/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:16.198084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=20a8f3dcbed2483a9503809231ceb898, user_id=c0781f1357314dd29217c8a1c21a4b5a, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:16.201342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:16.211267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1103/2200] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:16 2022] PUT /identity/v3/projects/20a8f3dcbed2483a9503809231ceb898/users/c0781f1357314dd29217c8a1c21a4b5a/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:16.276590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:16.276976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:16.277246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:16.306981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] MFA Rules not processed for user `c0781f1357314dd29217c8a1c21a4b5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:16.308809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:16.342643 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:16.383279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1098/2201] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:16 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:16.394963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:16.396749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:16.452819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee3c76d-b797-489e-9171-b8ec6bb22b7d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:16.456375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d7c434-ab5b-468f-8650-020ae0e4bcdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:16.456992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7c434-ab5b-468f-8650-020ae0e4bcdf service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:16.457263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7c434-ab5b-468f-8650-020ae0e4bcdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:16.457526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7c434-ab5b-468f-8650-020ae0e4bcdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:16.466245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d7c434-ab5b-468f-8650-020ae0e4bcdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:16.467648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d7c434-ab5b-468f-8650-020ae0e4bcdf service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:16.477468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1104/2202] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:16 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:17.016894 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:17.077759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690ad0e1-9b09-457b-b7aa-420d24b70295 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:17.082272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:17.082890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:17.083168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:17.083441 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] PATH_INFO: `/v3/users/f6d388da6e304502ab21382ec735001a` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:17.084550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6d388da6e304502ab21382ec735001a)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:17.086573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:17.132154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6d388da6e304502ab21382ec735001a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.144863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6d388da6e304502ab21382ec735001a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.145526 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6d388da6e304502ab21382ec735001a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.152527 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6d388da6e304502ab21382ec735001a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.153362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6d388da6e304502ab21382ec735001a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.165787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1099/2203] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:17 2022] DELETE /identity/v3/users/f6d388da6e304502ab21382ec735001a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:17.170826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:17.171312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:17.171431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:17.190134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:17.191328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:17.248211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1105/2204] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:17.257818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:17.259206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:17.306775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:17.319247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875f0945-e90e-409a-98ca-a74f263221a0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:17.322937 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:17.324059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:17.324333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:17.324598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:17.333457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:17.334882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:17.344823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1100/2205] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:17.619761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:17.621969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:17.634995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6eb7b0b-6ab2-40b0-ae32-3549f7fa140e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:17.641100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:17.641826 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:17.642001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:17.642266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] PATH_INFO: `/v3/projects/38377b9a9a6a4c75be150b4089bf7a73` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:17.643546 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] RBAC: Authorizing `identity:delete_project(project_id=38377b9a9a6a4c75be150b4089bf7a73)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:17.645483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:17.695375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38377b9a9a6a4c75be150b4089bf7a73', 'request_id': 'req-6973cd86-1da6-4efc-b8aa-1d755b85b26f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.700993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:17.702484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38377b9a9a6a4c75be150b4089bf7a73', 'request_id': 'req-6973cd86-1da6-4efc-b8aa-1d755b85b26f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.712041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38377b9a9a6a4c75be150b4089bf7a73', 'request_id': 'req-6973cd86-1da6-4efc-b8aa-1d755b85b26f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.724007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] The token cache is being invalidate because project 38377b9a9a6a4c75be150b4089bf7a73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:17.724297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.724972 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.726509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1106/2206] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:17 2022] DELETE /identity/v3/projects/38377b9a9a6a4c75be150b4089bf7a73 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:17.730490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:17.740235 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667709ea-3799-4e02-8893-a48e330a72bd service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:17.803206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:17.803999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:17.804342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:17.804535 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] PATH_INFO: `/v3/users/c0781f1357314dd29217c8a1c21a4b5a` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:17.805759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0781f1357314dd29217c8a1c21a4b5a)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:17.807688 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:17.859707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0781f1357314dd29217c8a1c21a4b5a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.873917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0781f1357314dd29217c8a1c21a4b5a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.874696 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0781f1357314dd29217c8a1c21a4b5a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.882355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0781f1357314dd29217c8a1c21a4b5a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.883057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-9061b03f-5881-4e9d-94a4-99da59b8b376 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0781f1357314dd29217c8a1c21a4b5a'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:17.897262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1101/2207] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:17 2022] DELETE /identity/v3/users/c0781f1357314dd29217c8a1c21a4b5a => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:18.168243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.169983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:18.184444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6973cd86-1da6-4efc-b8aa-1d755b85b26f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.243462 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.244104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.244383 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.244861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] PATH_INFO: `/v3/projects/20a8f3dcbed2483a9503809231ceb898` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.246472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20a8f3dcbed2483a9503809231ceb898)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.248382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.295632 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20a8f3dcbed2483a9503809231ceb898', 'request_id': 'req-61c21602-ba41-4afa-becf-9cb675f99989'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:18.301226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20a8f3dcbed2483a9503809231ceb898', 'request_id': 'req-61c21602-ba41-4afa-becf-9cb675f99989'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:18.309172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20a8f3dcbed2483a9503809231ceb898', 'request_id': 'req-61c21602-ba41-4afa-becf-9cb675f99989'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:18.320984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] The token cache is being invalidate because project 20a8f3dcbed2483a9503809231ceb898 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:18.321274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:18.322021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-61c21602-ba41-4afa-becf-9cb675f99989 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:18.323528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1107/2208] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:18 2022] DELETE /identity/v3/projects/20a8f3dcbed2483a9503809231ceb898 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:18.328483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.329016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.329016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.348737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:18.349924 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:18.410679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1102/2209] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:18.419215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.419501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.419799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.439696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:18.440975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:18.449123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1108/2210] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:18.453338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.462956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d3206d-b7be-4ee8-85bd-104c03027f3a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.468048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.468640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.468961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.469233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.469928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:18.470228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.471874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.473343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:18.486662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1103/2211] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:36:18.490742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.500265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290807d2-c11d-4e58-a3ac-52260ab9a424 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.504922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.505515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.505786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.506053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.506685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.508567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.536257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1109/2212] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:18 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:18.540183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.550923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d593e483-fc8a-4fb0-aa22-a4760c7dc56c admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.613962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.614698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.614977 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.615248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.615923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.617873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.643168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1104/2213] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:18 2022] POST /identity/v3/users => generated 285 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:18.644770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.656473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a0031c-b01e-459f-8ea3-05fc1231759f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.661257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.661861 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.662134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.662405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.663046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.664630 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.675713 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1110/2214] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:18.680190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.693072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a6ee5c-9621-4a1d-8cfd-c1645e521690 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.698943 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f92995-7509-4466-a8de-c58a1fd53887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.699857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f92995-7509-4466-a8de-c58a1fd53887 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.700137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f92995-7509-4466-a8de-c58a1fd53887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.700521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f92995-7509-4466-a8de-c58a1fd53887 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.701218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f92995-7509-4466-a8de-c58a1fd53887 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.703800 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f92995-7509-4466-a8de-c58a1fd53887 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.719906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1105/2215] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:18 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:18.721939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.732357 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258268d5-ae19-4f62-b302-b9c0936d504d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.737008 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.737659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.737935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.738206 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] PATH_INFO: `/v3/projects/cfda6acef92b48799f37336434e772e0/users/7423a07995044ea2b61978b4927baf21/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.754034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfda6acef92b48799f37336434e772e0, user_id=7423a07995044ea2b61978b4927baf21, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.756688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.771404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1111/2216] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:18 2022] PUT /identity/v3/projects/cfda6acef92b48799f37336434e772e0/users/7423a07995044ea2b61978b4927baf21/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:18.782066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.782412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.782690 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.810772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] MFA Rules not processed for user `7423a07995044ea2b61978b4927baf21`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:18.811925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:18.881298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1106/2217] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:18.895511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:18.897374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:18.913097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:18.965494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0a39e-b56f-4bd0-bbed-38756dbb96ff admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:18.974028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d2788633742480981376ed8b171ac89', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:18.974733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:18.975079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:18.975450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:18.985535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:18.987136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:18.997376 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1112/2218] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:18 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:28.683379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:28.684807 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:28.742399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ada20-c05f-4d91-885c-81fc7ba12f10 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:28.746106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:28.746706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:28.747096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:28.747367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:28.749464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:28.764442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:28.815314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:28.816722 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:28.827033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1107/2219] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:28.992944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=21 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:32.397585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:32.399248 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:32.415931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8d6d8f-4aab-4754-a436-ca66045a1c4c service glance] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:32.420431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:32.421075 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:32.421347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:32.421613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] PATH_INFO: `/v3/users/1fceec891ef24ba3a39da72c90fb733a` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:32.422748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=1fceec891ef24ba3a39da72c90fb733a)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:32.424648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:32.470984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1fceec891ef24ba3a39da72c90fb733a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.483054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1fceec891ef24ba3a39da72c90fb733a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.483718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1fceec891ef24ba3a39da72c90fb733a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.490739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1fceec891ef24ba3a39da72c90fb733a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.491345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5c559bb9-8964-4097-bf8b-ce323630f7dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1fceec891ef24ba3a39da72c90fb733a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.503316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1113/2220] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:32 2022] DELETE /identity/v3/users/1fceec891ef24ba3a39da72c90fb733a => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:32.675751 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:32.686320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf41148-3794-4ceb-8cd7-153562ff2d0a service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:32.746330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:32.746959 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:32.747238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:32.747505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] PATH_INFO: `/v3/projects/76e921b067f04677b76c245b5f93e112` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:32.748619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76e921b067f04677b76c245b5f93e112)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:32.750675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:32.798130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76e921b067f04677b76c245b5f93e112', 'request_id': 'req-929146e5-a65a-4265-9932-604103c8a494'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.804017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76e921b067f04677b76c245b5f93e112', 'request_id': 'req-929146e5-a65a-4265-9932-604103c8a494'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.811330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76e921b067f04677b76c245b5f93e112', 'request_id': 'req-929146e5-a65a-4265-9932-604103c8a494'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.825270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] The token cache is being invalidate because project 76e921b067f04677b76c245b5f93e112 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:32.825554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.826178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-929146e5-a65a-4265-9932-604103c8a494 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:32.827757 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1108/2221] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:32 2022] DELETE /identity/v3/projects/76e921b067f04677b76c245b5f93e112 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:32.832623 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:32.832937 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:32.833209 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:32.852930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:32.854130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:32.914438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1114/2222] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:32.922603 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:32.922903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:32.923174 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:32.943228 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:32.944427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:32.952234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1109/2223] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:32.956260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:32.965833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c0489f-64d1-4366-9f7f-ac92daf29854 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:32.970442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:32.971054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:32.971329 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:32.971597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:32.972230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:32.972526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:32.974173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:32.986287 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1115/2224] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:36:32.990616 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:33.000315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b667fa-f95e-4e0f-9a65-64edb93fc8cb None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:33.005085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:33.005671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.005941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.006206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.006828 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:33.008741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:33.034522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1110/2225] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:32 2022] POST /identity/v3/projects => generated 365 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:33.038570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:33.048779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926f45b-5c91-41e5-a7e3-01211dc55603 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:33.111853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:33.112461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.112762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.113032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.113776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:33.115575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:33.124379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:33.142219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1116/2226] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:33 2022] POST /identity/v3/users => generated 283 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:33.143641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:33.154094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ecb1f-c027-452f-afa2-93c16a474b7d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:33.158853 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:33.159457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.159732 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.160001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.160640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:33.162272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:33.173374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1111/2227] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:33.177378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:33.188346 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac71fc-f403-40d3-aa9d-03ff11ec4caf admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:33.193643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f0376a-8597-4980-a03f-ffbf5ce5db15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:33.194469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f0376a-8597-4980-a03f-ffbf5ce5db15 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.194744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f0376a-8597-4980-a03f-ffbf5ce5db15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.195028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f0376a-8597-4980-a03f-ffbf5ce5db15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.195653 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f0376a-8597-4980-a03f-ffbf5ce5db15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:33.197290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f0376a-8597-4980-a03f-ffbf5ce5db15 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:33.209582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1117/2228] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:33.213342 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:33.220163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:33.226060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4162ca1-78f2-40bb-a849-bc90dd893bfe admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:33.235766 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:33.236923 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.237208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.237668 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] PATH_INFO: `/v3/projects/08b94d09445e43e5a01b4cf9db247a3f/users/2f5f5a0a2d614eff9cf22858abf108ad/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.259202 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08b94d09445e43e5a01b4cf9db247a3f, user_id=2f5f5a0a2d614eff9cf22858abf108ad, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:33.262060 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:33.298667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1112/2229] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:33 2022] PUT /identity/v3/projects/08b94d09445e43e5a01b4cf9db247a3f/users/2f5f5a0a2d614eff9cf22858abf108ad/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:33.305299 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.305581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.305860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.334830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] MFA Rules not processed for user `2f5f5a0a2d614eff9cf22858abf108ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:33.336170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:33.416845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1118/2230] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:33 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:33.425268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:33.427029 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:33.486443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a86ae-0c7d-45a1-8651-0fce73426db3 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:33.490367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:33.490969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:33.491244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:33.491510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:33.499989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:33.501481 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:33.510126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1113/2231] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:33 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:35.869444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:35.871290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:35.927661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ad5148-cc5a-4895-a656-5d15dfb08f52 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:35.931227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:35.931864 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:35.932139 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:35.932443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:35.934733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:35.997031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:35.998484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a95cbb-f736-43b2-b5a8-fba040ce2412 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:36.008456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1119/2232] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:35 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:43.506163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=35,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:46.003984 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=3,keystone:SELECT=89 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:48.420226 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:48.484332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113942f6-41b5-48d7-b990-2ffa0fae8512 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:48.488773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:48.489399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:48.489673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:48.490047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] PATH_INFO: `/v3/users/2f5f5a0a2d614eff9cf22858abf108ad` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:48.491169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f5f5a0a2d614eff9cf22858abf108ad)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:48.493140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:48.537919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f5f5a0a2d614eff9cf22858abf108ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:48.550354 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f5f5a0a2d614eff9cf22858abf108ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:48.551017 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f5f5a0a2d614eff9cf22858abf108ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:48.558261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f5f5a0a2d614eff9cf22858abf108ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:48.558876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f5f5a0a2d614eff9cf22858abf108ad'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:48.570758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1114/2233] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:48 2022] DELETE /identity/v3/users/2f5f5a0a2d614eff9cf22858abf108ad => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:48.575253 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:48.575701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:48.575797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:48.595050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:48.596239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:48.654636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1120/2234] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:48.664194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:48.665628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:48.722034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6cbf3d-1dea-48e9-ad95-fc9c6600dd3b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:48.725682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ecfa1e-48c4-4972-9bc9-046e587b752b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:48.726276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ecfa1e-48c4-4972-9bc9-046e587b752b service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:48.726572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ecfa1e-48c4-4972-9bc9-046e587b752b service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:48.726843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ecfa1e-48c4-4972-9bc9-046e587b752b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:48.736542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ecfa1e-48c4-4972-9bc9-046e587b752b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:48.738021 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ecfa1e-48c4-4972-9bc9-046e587b752b service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:48.748115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1115/2235] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:49.011441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.012874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:49.026914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1a877-0290-4700-97a5-0f2b318048c3 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.031635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.032286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.032566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.032864 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] PATH_INFO: `/v3/projects/08b94d09445e43e5a01b4cf9db247a3f` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.033991 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08b94d09445e43e5a01b4cf9db247a3f)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.035890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.082150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08b94d09445e43e5a01b4cf9db247a3f', 'request_id': 'req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:49.087644 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08b94d09445e43e5a01b4cf9db247a3f', 'request_id': 'req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:49.096752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08b94d09445e43e5a01b4cf9db247a3f', 'request_id': 'req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:49.109397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] The token cache is being invalidate because project 08b94d09445e43e5a01b4cf9db247a3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:49.109682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:49.110333 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a4a0d5f4-d2e2-4e26-ad16-7b5cac5baee0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:49.111833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1121/2236] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:49 2022] DELETE /identity/v3/projects/08b94d09445e43e5a01b4cf9db247a3f => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:49.116730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.117115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.117451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.137464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:49.138648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:49.155252 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:49.197433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1116/2237] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:49.205585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.205893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.206162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.226873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:49.228066 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:49.236793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1122/2238] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:49.240982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.251739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dcbb86-e4b4-4339-b273-d3927c8b4b64 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.256306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.256930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.258023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.258299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.258947 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:49.259242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.261142 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.277617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1117/2239] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:36:49.278740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.288529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570b3df-d800-496d-a891-7fdde54c102a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.294128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.294735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.295010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.295279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.295950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.298194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.331222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1123/2240] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/projects => generated 403 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:49.335593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.346568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb167dc-fd99-4e4f-95b0-61029474d58d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.409948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.410548 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.410826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.411096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.411727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.413694 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.435513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1118/2241] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/users => generated 302 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:49.439719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.450627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c98866-25ac-4e92-b282-558c4760a52e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.455311 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace8359c-5580-444e-a155-f358716caece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.455903 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace8359c-5580-444e-a155-f358716caece admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.456178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace8359c-5580-444e-a155-f358716caece admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.456445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace8359c-5580-444e-a155-f358716caece admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.457111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace8359c-5580-444e-a155-f358716caece admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.458834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace8359c-5580-444e-a155-f358716caece admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.471365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1124/2242] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:49.475261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.485534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49181b9-2c89-419d-a53e-e43e92ab9134 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.490127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a49ebd-f237-44df-9d98-34476faa31b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.490717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a49ebd-f237-44df-9d98-34476faa31b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.490991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a49ebd-f237-44df-9d98-34476faa31b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.491257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a49ebd-f237-44df-9d98-34476faa31b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.491891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a49ebd-f237-44df-9d98-34476faa31b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.493534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a49ebd-f237-44df-9d98-34476faa31b6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.503982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1119/2243] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:49.507983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace8359c-5580-444e-a155-f358716caece admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.518306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace8359c-5580-444e-a155-f358716caece admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.523042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.523684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.523959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.524229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] PATH_INFO: `/v3/projects/837c889c8ee94eba815951c7545697ad/users/ee1a7006e728429b99e405225e882ee1/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.539620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=837c889c8ee94eba815951c7545697ad, user_id=ee1a7006e728429b99e405225e882ee1, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.542541 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.555418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1125/2244] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:49 2022] PUT /identity/v3/projects/837c889c8ee94eba815951c7545697ad/users/ee1a7006e728429b99e405225e882ee1/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:49.565704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.565989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.566259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.596213 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] MFA Rules not processed for user `ee1a7006e728429b99e405225e882ee1`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:49.597478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:49.661414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1120/2245] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:49.666221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.686590 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:49.729708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f723728-1141-4010-b789-90a428809b77 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.734821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.735418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.735689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.735954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.736584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.738500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.764963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1126/2246] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/projects => generated 405 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:49.769145 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.780758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b82005-90c9-4000-b508-6fd62efeaeb1 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.793331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:49.838564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.839163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.839436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.839704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.840324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.842308 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.863249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1121/2247] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:49 2022] POST /identity/v3/users => generated 309 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:49.867241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.877809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3808185-6ec9-40d8-9d88-1742fd1f95cc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.882331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.882918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.883187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.883449 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.884069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.886201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.896579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1127/2248] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:49.900331 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.910362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7a4d4c-7e9d-4e17-844e-3218d3eba1b5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:49.914973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:49.915610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:49.915892 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:49.916159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] PATH_INFO: `/v3/projects/f821c68e8d4d4670b2fddf4cd4b6fea2/users/ea36d356007c4b418d667c2374a01459/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:49.930587 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f821c68e8d4d4670b2fddf4cd4b6fea2, user_id=ea36d356007c4b418d667c2374a01459, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:49.933908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:49.946199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1122/2249] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:49 2022] PUT /identity/v3/projects/f821c68e8d4d4670b2fddf4cd4b6fea2/users/ea36d356007c4b418d667c2374a01459/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:49.950697 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:49.960458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74dc52c-d345-4dcc-9473-f721741fb3d4 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:50.015908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928ea5c6-b5ca-4ad4-9a50-ebacff339aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:50.016446 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928ea5c6-b5ca-4ad4-9a50-ebacff339aaf admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.016746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928ea5c6-b5ca-4ad4-9a50-ebacff339aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.017020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928ea5c6-b5ca-4ad4-9a50-ebacff339aaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.017679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928ea5c6-b5ca-4ad4-9a50-ebacff339aaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:50.019259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928ea5c6-b5ca-4ad4-9a50-ebacff339aaf admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:50.028840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1128/2250] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:50.033998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:50.044089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547a9acd-fb38-4dde-a86c-7d72b95e058b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:50.048554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:50.049222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.049494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.049779 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] PATH_INFO: `/v3/projects/f821c68e8d4d4670b2fddf4cd4b6fea2/users/ea36d356007c4b418d667c2374a01459/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.051625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f821c68e8d4d4670b2fddf4cd4b6fea2, user_id=ea36d356007c4b418d667c2374a01459, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:50.054283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:50.064769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1123/2251] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:50 2022] PUT /identity/v3/projects/f821c68e8d4d4670b2fddf4cd4b6fea2/users/ea36d356007c4b418d667c2374a01459/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:50.076760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.077044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.077314 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.104733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] MFA Rules not processed for user `ea36d356007c4b418d667c2374a01459`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:50.106184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:50.173873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1129/2252] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:50 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:50.185143 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:50.186848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:50.242662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea2e920-4fa9-4abd-a0ee-663a553686f4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:50.246903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58349544-2bd9-4a6c-8cad-91346c975ca0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:50.247488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58349544-2bd9-4a6c-8cad-91346c975ca0 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.247761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58349544-2bd9-4a6c-8cad-91346c975ca0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.248028 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58349544-2bd9-4a6c-8cad-91346c975ca0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.256907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58349544-2bd9-4a6c-8cad-91346c975ca0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:50.258333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58349544-2bd9-4a6c-8cad-91346c975ca0 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:50.267727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1124/2253] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:50 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:50.429051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:50.449170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:50.501998 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20619c3-9b20-40c1-a427-7d51fc41ae93 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:50.506516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:50.507194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.507468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.507751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] PATH_INFO: `/v3/users/ee1a7006e728429b99e405225e882ee1` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.508894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee1a7006e728429b99e405225e882ee1)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:50.510960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:50.556180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee1a7006e728429b99e405225e882ee1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:50.567939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee1a7006e728429b99e405225e882ee1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:50.568604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee1a7006e728429b99e405225e882ee1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:50.575987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee1a7006e728429b99e405225e882ee1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:50.576664 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee1a7006e728429b99e405225e882ee1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:50.589220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1130/2254] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:50 2022] DELETE /identity/v3/users/ee1a7006e728429b99e405225e882ee1 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:50.594041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.594514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.594610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.613922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:50.615104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:50.673343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1125/2255] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:50.683301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:50.685078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:50.745712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39558a40-79e5-48db-82c3-b63460c35745 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:50.749441 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:50.750049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:50.750324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:50.750591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:50.759477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:50.760916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:50.770438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1131/2256] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:51.032951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.034368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.047849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4707bc4-b9b3-4101-b798-42267653bdf6 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.052872 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.053671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.053955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.054311 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] PATH_INFO: `/v3/projects/837c889c8ee94eba815951c7545697ad` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.055337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=837c889c8ee94eba815951c7545697ad)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.057333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.065781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:51.104316 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '837c889c8ee94eba815951c7545697ad', 'request_id': 'req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.110414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '837c889c8ee94eba815951c7545697ad', 'request_id': 'req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.117013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '837c889c8ee94eba815951c7545697ad', 'request_id': 'req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.130162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] The token cache is being invalidate because project 837c889c8ee94eba815951c7545697ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:51.130449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.131092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.132576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1126/2257] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:51 2022] DELETE /identity/v3/projects/837c889c8ee94eba815951c7545697ad => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:51.137070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.146523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af5bf6b-389b-4e04-8e6d-7260168abfa9 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.179018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:51.206406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.207037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.207316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.207585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] PATH_INFO: `/v3/users/ea36d356007c4b418d667c2374a01459` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.209266 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea36d356007c4b418d667c2374a01459)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.210709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.255833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea36d356007c4b418d667c2374a01459'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.268280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea36d356007c4b418d667c2374a01459'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.268971 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea36d356007c4b418d667c2374a01459'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.275509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea36d356007c4b418d667c2374a01459'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.276111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea36d356007c4b418d667c2374a01459'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.287865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1132/2258] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:51 2022] DELETE /identity/v3/users/ea36d356007c4b418d667c2374a01459 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:51.530999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.532933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.547162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e04f6c-7e7b-4255-8101-1fba6cdb9a13 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.552135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.553811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.611349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.612073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.612484 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.612691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] PATH_INFO: `/v3/users/7423a07995044ea2b61978b4927baf21` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.613841 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7423a07995044ea2b61978b4927baf21)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.615896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.629619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9483fc-954c-4118-9bc3-0840095bc332 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.634205 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.634757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.635028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.635820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] PATH_INFO: `/v3/projects/f821c68e8d4d4670b2fddf4cd4b6fea2` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.636941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f821c68e8d4d4670b2fddf4cd4b6fea2)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.638821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.663225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7423a07995044ea2b61978b4927baf21'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.677161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7423a07995044ea2b61978b4927baf21'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.677860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7423a07995044ea2b61978b4927baf21'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.687898 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7423a07995044ea2b61978b4927baf21'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.688059 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f821c68e8d4d4670b2fddf4cd4b6fea2', 'request_id': 'req-f2f24697-c468-454a-88c5-f21326ac3850'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.688781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-998b72a9-86bc-4cda-adf2-ea138bc32c48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7423a07995044ea2b61978b4927baf21'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.693846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f821c68e8d4d4670b2fddf4cd4b6fea2', 'request_id': 'req-f2f24697-c468-454a-88c5-f21326ac3850'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.695473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:51.700873 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f821c68e8d4d4670b2fddf4cd4b6fea2', 'request_id': 'req-f2f24697-c468-454a-88c5-f21326ac3850'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.702515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1127/2259] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:36:51 2022] DELETE /identity/v3/users/7423a07995044ea2b61978b4927baf21 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:51.706826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.707111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.707406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.714045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] The token cache is being invalidate because project f821c68e8d4d4670b2fddf4cd4b6fea2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:51.714463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.715106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f2f24697-c468-454a-88c5-f21326ac3850 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:51.719866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1133/2260] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:51 2022] DELETE /identity/v3/projects/f821c68e8d4d4670b2fddf4cd4b6fea2 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:51.721231 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ae6da-eb02-4658-a691-5abde1b29fd4 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.721502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ae6da-eb02-4658-a691-5abde1b29fd4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.721765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ae6da-eb02-4658-a691-5abde1b29fd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.728080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:51.729842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.739847 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:51.743058 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-979ae6da-eb02-4658-a691-5abde1b29fd4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:51.744224 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-979ae6da-eb02-4658-a691-5abde1b29fd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.791255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1128/2261] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:51.802018 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.803393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.808744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1134/2262] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:51.816244 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.816600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.816916 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.837870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:51.839103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:51.847156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1135/2263] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:51.851225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.860895 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454699ab-aa7e-4e4a-8db4-f6ad0b5e324a None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.863088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33517477-940c-4b4b-a8b2-cb3b2c991947 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.865425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.866004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.866278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.866543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.866657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.867179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:51.867289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.867465 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.867571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.867737 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.869093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.877412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.879388 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.884576 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1136/2264] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:36:51.885512 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.889680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1129/2265] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:51.894779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4348f8dd-f737-44eb-873b-0fed724ec81b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:51.899530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:51.900124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:51.900398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:51.900693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:51.901379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:51.903252 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:51.933460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1137/2266] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:51 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:51.938154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:51.949045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd3e40-48dc-4ad6-872a-03394c06144e service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.010759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.011387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.011657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.011920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.012543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.014498 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.040743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1130/2267] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:51 2022] POST /identity/v3/users => generated 291 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:52.042994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.053987 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96bc245-27c1-410b-ab02-c022e8646b77 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.058635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.059240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.059514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.059779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.060419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.062044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.072181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1138/2268] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:52.076081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.086618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e152bbe9-a6f2-4403-8895-24ec5cc7fd44 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.091857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.092459 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.092760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.093038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.093680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.095286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.109329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1131/2269] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:52.113860 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.124009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ace207-5479-4081-a66c-cbf3d88a83a2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.129163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f2ef24-2346-433a-9703-24e1bfabe772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.129898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f2ef24-2346-433a-9703-24e1bfabe772 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.130180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f2ef24-2346-433a-9703-24e1bfabe772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.130455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f2ef24-2346-433a-9703-24e1bfabe772 admin admin] PATH_INFO: `/v3/projects/373ae7ea071848548aafda02674c4cfc/users/a5b8873b972c4ecba5c8032f2481e2c5/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.145897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f2ef24-2346-433a-9703-24e1bfabe772 admin admin] RBAC: Authorizing `identity:create_grant(project_id=373ae7ea071848548aafda02674c4cfc, user_id=a5b8873b972c4ecba5c8032f2481e2c5, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.148523 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f2ef24-2346-433a-9703-24e1bfabe772 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.161996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1139/2270] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:52 2022] PUT /identity/v3/projects/373ae7ea071848548aafda02674c4cfc/users/a5b8873b972c4ecba5c8032f2481e2c5/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:52.170666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.172801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.173298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.173642 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.173977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.188764 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f64a660-c582-4d62-bc23-0a7d8c26d242 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.207165 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] MFA Rules not processed for user `a5b8873b972c4ecba5c8032f2481e2c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:52.208457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.225554 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:52.250824 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.251591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.251970 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.252364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] PATH_INFO: `/v3/projects/cfda6acef92b48799f37336434e772e0` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.253653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfda6acef92b48799f37336434e772e0)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.255695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.272009 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:INSERT=3,keystone:DELETE=1,keystone:SELECT=96 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:52.279586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1140/2271] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:36:52.280448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.292366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703dd768-5dcd-47cc-811b-c7631ac50af6 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.304686 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfda6acef92b48799f37336434e772e0', 'request_id': 'req-807c4068-c751-4be8-91c8-9c817e262681'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:52.310525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfda6acef92b48799f37336434e772e0', 'request_id': 'req-807c4068-c751-4be8-91c8-9c817e262681'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:52.317324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfda6acef92b48799f37336434e772e0', 'request_id': 'req-807c4068-c751-4be8-91c8-9c817e262681'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:52.332085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] The token cache is being invalidate because project cfda6acef92b48799f37336434e772e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:36:52.332366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:52.333386 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-807c4068-c751-4be8-91c8-9c817e262681 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:36:52.335282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1132/2272] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:36:52 2022] DELETE /identity/v3/projects/cfda6acef92b48799f37336434e772e0 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:36:52.340519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bf0ced-7584-4ee8-8f92-26ec3ef8bd51 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.340823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bf0ced-7584-4ee8-8f92-26ec3ef8bd51 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.341378 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bf0ced-7584-4ee8-8f92-26ec3ef8bd51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.350124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.350731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.351014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.351279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.351913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.353831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.361445 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-d3bf0ced-7584-4ee8-8f92-26ec3ef8bd51 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:52.362645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d3bf0ced-7584-4ee8-8f92-26ec3ef8bd51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.370505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1133/2273] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:52.378556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.378862 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.379104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.381539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1141/2274] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/projects => generated 381 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:36:52.385931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.387702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.402634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:52.403719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.450956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b91935-46db-4811-b523-23b458dbc078 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.455601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.456211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.456485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.456784 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.457530 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.459437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.464197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1134/2275] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:52.468464 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.477279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da51f0e7-6757-4a46-b264-3fa7d2692288 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.482151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.485668 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1142/2276] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/users => generated 297 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:52.485825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.485825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.485825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.485825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:36:52.485825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.485825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.487393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.496915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0c91dc-40e0-48c5-ab21-6f444c30f386 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.500723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1135/2277] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 12 02:36:52.502049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.504884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.505482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.505760 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.506033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.506699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.508485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.512350 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f47f952-f787-4f28-a80a-cc53e455e5bf admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.516996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.517569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.517848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.518114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.518726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.520610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.521268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1143/2278] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:52.525113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.535648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c87228d-bd93-4608-ae56-85df31bd6019 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.540431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.541162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.541444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.541718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] PATH_INFO: `/v3/projects/0ce58f84719c4943ad1bf4074b3dcd88/users/03085d1158b648318edb61a8171353a3/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.548933 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1136/2279] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:52.552692 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.560327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ce58f84719c4943ad1bf4074b3dcd88, user_id=03085d1158b648318edb61a8171353a3, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.563041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.576199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1144/2280] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:52 2022] PUT /identity/v3/projects/0ce58f84719c4943ad1bf4074b3dcd88/users/03085d1158b648318edb61a8171353a3/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:52.580491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.584594 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:52.619865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5da68e-8b59-43bc-8f46-b98b29678fc4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.623751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:52.645055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a6b28a-9a8d-4acf-8bc9-02e5b9347cbc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.650358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.651621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.651905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.652222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.652958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.654496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.664491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1145/2281] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:52.668539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.676993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dba122-c8b0-45ca-a852-c9a81b667be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.677681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dba122-c8b0-45ca-a852-c9a81b667be4 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.677957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dba122-c8b0-45ca-a852-c9a81b667be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.678225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dba122-c8b0-45ca-a852-c9a81b667be4 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.678525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70426586-a519-49de-97a9-b3d0e6087af6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.678851 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dba122-c8b0-45ca-a852-c9a81b667be4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.680821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dba122-c8b0-45ca-a852-c9a81b667be4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.683276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.683910 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.684255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.684526 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] PATH_INFO: `/v3/projects/0ce58f84719c4943ad1bf4074b3dcd88/users/03085d1158b648318edb61a8171353a3/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.686459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ce58f84719c4943ad1bf4074b3dcd88, user_id=03085d1158b648318edb61a8171353a3, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.689401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.700490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1146/2282] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:52 2022] PUT /identity/v3/projects/0ce58f84719c4943ad1bf4074b3dcd88/users/03085d1158b648318edb61a8171353a3/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:52.708681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1137/2283] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/users => generated 283 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:36:52.710761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.711789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.712067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.712409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.741754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] MFA Rules not processed for user `03085d1158b648318edb61a8171353a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:52.742982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.774156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b9434-1bf4-4a8f-82f0-05228c52c2b8 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.779229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.779833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.780104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.780371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.781029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.782616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.792720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1147/2284] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:36:52.796545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.805823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75badc24-ec3c-44eb-ae12-c9e709d4f76d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.811129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.811702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.811970 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.812233 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.812862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.814409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.816381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1138/2285] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:52.827404 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1148/2286] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:36:52.829188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.830655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.831442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.841737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c277e-14e1-4038-a779-73cb13b3ca9b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.846445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656a6550-eddb-4107-b1ad-4b6d4c5e8e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.847088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656a6550-eddb-4107-b1ad-4b6d4c5e8e78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.847360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656a6550-eddb-4107-b1ad-4b6d4c5e8e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.847629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656a6550-eddb-4107-b1ad-4b6d4c5e8e78 admin admin] PATH_INFO: `/v3/projects/048b0e277eab4dcf94114c329e8263ab/users/b8cff3ae919549ef8cd6d18650903fbf/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.862070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656a6550-eddb-4107-b1ad-4b6d4c5e8e78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=048b0e277eab4dcf94114c329e8263ab, user_id=b8cff3ae919549ef8cd6d18650903fbf, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:52.864724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656a6550-eddb-4107-b1ad-4b6d4c5e8e78 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:52.878173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1149/2287] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:36:52 2022] PUT /identity/v3/projects/048b0e277eab4dcf94114c329e8263ab/users/b8cff3ae919549ef8cd6d18650903fbf/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:36:52.883747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:52.888788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd3cc97-1d01-4f2d-b759-4d40faa2810a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:52.889234 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.889516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.889810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.917693 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] MFA Rules not processed for user `b8cff3ae919549ef8cd6d18650903fbf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:36:52.918806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:52.939295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:52.939905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:52.940179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:52.940444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:52.982904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1150/2288] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:36:52 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:36:52.994511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:52.996374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:53.003999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:53.005473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:53.007904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f69cf82-4192-4d17-b640-436dfc777003 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:53.012118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:53.012735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:53.013013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:53.013280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:53.015424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1139/2289] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:36:53.021853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:53.023260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:53.027795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:36:53.032807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1151/2290] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:52 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:36:53.092887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:36:53.107273 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310d0e5-83d1-4eea-b771-f3d928aeeac0 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:36:53.111009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:36:53.111612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:36:53.111999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:36:53.112282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:36:53.114065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:36:53.174727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:36:53.176078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b96c156-36a0-4c7c-a5a8-bf983bb4fdcd service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:36:53.186299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1140/2291] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:36:53 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:02.225647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:02.302732 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5731fc6-694e-4f07-b4b7-bebab094bd0b service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:02.313438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:02.314087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:02.314358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:02.314631 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] PATH_INFO: `/v3/users/a5b8873b972c4ecba5c8032f2481e2c5` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:02.315757 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5b8873b972c4ecba5c8032f2481e2c5)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:02.317715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:02.368067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5b8873b972c4ecba5c8032f2481e2c5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.380982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5b8873b972c4ecba5c8032f2481e2c5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.381753 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5b8873b972c4ecba5c8032f2481e2c5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.389559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5b8873b972c4ecba5c8032f2481e2c5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.390483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5b8873b972c4ecba5c8032f2481e2c5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.407038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1152/2292] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:02 2022] DELETE /identity/v3/users/a5b8873b972c4ecba5c8032f2481e2c5 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:02.411497 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:02.411954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:02.412069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:02.431097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:02.432893 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:02.482282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:02.495834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1141/2293] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:02.505673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:02.507153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:02.572065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759b23f8-0091-4f30-b3d8-f01b17593e34 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:02.575721 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:02.576325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:02.576598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:02.576896 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:02.585907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:02.587324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:02.597379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1153/2294] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:02.868016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:02.869483 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:02.882725 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca69a5-e133-44fd-8fe3-79d29b1d7a7d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:02.887795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:02.888466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:02.888762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:02.889030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] PATH_INFO: `/v3/projects/373ae7ea071848548aafda02674c4cfc` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:02.890141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=373ae7ea071848548aafda02674c4cfc)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:02.892179 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:02.938292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '373ae7ea071848548aafda02674c4cfc', 'request_id': 'req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.943686 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '373ae7ea071848548aafda02674c4cfc', 'request_id': 'req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.950542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '373ae7ea071848548aafda02674c4cfc', 'request_id': 'req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.962307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] The token cache is being invalidate because project 373ae7ea071848548aafda02674c4cfc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:02.962592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.963835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:02.965339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1142/2295] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:02 2022] DELETE /identity/v3/projects/373ae7ea071848548aafda02674c4cfc => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:02.969306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:02.978750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4999ad3-8432-4e3a-aa1f-838c8e8eb890 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:02.996269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:03.035236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:03.035857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.036130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.036398 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] PATH_INFO: `/v3/users/03085d1158b648318edb61a8171353a3` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.037517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03085d1158b648318edb61a8171353a3)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:03.039413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:03.083712 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03085d1158b648318edb61a8171353a3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.095736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03085d1158b648318edb61a8171353a3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.096393 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03085d1158b648318edb61a8171353a3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.103968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03085d1158b648318edb61a8171353a3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.104790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b24fb337-3dc9-405e-9ba7-b02c5e965e78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03085d1158b648318edb61a8171353a3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.118665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1154/2296] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:02 2022] DELETE /identity/v3/users/03085d1158b648318edb61a8171353a3 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:03.506524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:03.507993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:03.521865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b255bc-63aa-406a-bf2c-b05f8adaa6b3 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:03.580301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:03.580957 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.581236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.581508 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] PATH_INFO: `/v3/projects/0ce58f84719c4943ad1bf4074b3dcd88` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.582629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ce58f84719c4943ad1bf4074b3dcd88)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:03.584556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:03.633499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ce58f84719c4943ad1bf4074b3dcd88', 'request_id': 'req-3acd3812-3fff-490f-ae3b-e6d080410d3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.639758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ce58f84719c4943ad1bf4074b3dcd88', 'request_id': 'req-3acd3812-3fff-490f-ae3b-e6d080410d3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.646826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ce58f84719c4943ad1bf4074b3dcd88', 'request_id': 'req-3acd3812-3fff-490f-ae3b-e6d080410d3d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.660414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] The token cache is being invalidate because project 0ce58f84719c4943ad1bf4074b3dcd88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:03.660734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.661366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-3acd3812-3fff-490f-ae3b-e6d080410d3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:03.662886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1143/2297] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:03 2022] DELETE /identity/v3/projects/0ce58f84719c4943ad1bf4074b3dcd88 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:03.667797 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177acf79-cab7-49f9-97f5-148763279ade None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.668298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177acf79-cab7-49f9-97f5-148763279ade None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.668474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177acf79-cab7-49f9-97f5-148763279ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.688064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-177acf79-cab7-49f9-97f5-148763279ade None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:03.689384 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-177acf79-cab7-49f9-97f5-148763279ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:03.749883 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1155/2298] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:37:03.757782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.758066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.758393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.773296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:03.782157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:03.784052 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:03.798071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1144/2299] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:03.802394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177acf79-cab7-49f9-97f5-148763279ade None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:03.812490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177acf79-cab7-49f9-97f5-148763279ade None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:03.817214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:03.817922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.818301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.818570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.819247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:03.819503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:03.821946 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:03.835330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1156/2300] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:03.840541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:03.850672 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdd3bdd-1f23-4dd5-a933-a1fc4b0552a7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:03.856034 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:03.856789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.857250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.857592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.858306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:03.860299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:03.892323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1145/2301] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:03 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:03.896220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:03.907152 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be14330-6630-4ffe-a976-e258b923224e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:03.935148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:03.968296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:03.968938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:03.969215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:03.969622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:03.970115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:03.972005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:03.993257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1157/2302] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:03 2022] POST /identity/v3/users => generated 283 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:03.998020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.008368 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c5be68-29bf-4aac-857b-85d9e8c5b9d8 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.013691 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.014279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.014553 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.014819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.015453 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.017092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.026918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1146/2303] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:04.030645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.040291 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c18a599-aeb8-4fe2-8750-cc52148818b3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.045063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb5f5ba-9c3a-4a81-911f-4210e02600ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.045651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb5f5ba-9c3a-4a81-911f-4210e02600ef admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.045928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb5f5ba-9c3a-4a81-911f-4210e02600ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.046215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb5f5ba-9c3a-4a81-911f-4210e02600ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.046824 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb5f5ba-9c3a-4a81-911f-4210e02600ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.048397 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb5f5ba-9c3a-4a81-911f-4210e02600ef admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.059455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1158/2304] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:37:04.063509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.074798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31bd6-38f1-4004-a7a6-117e2adaba7a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.081134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.081774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.082046 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.082310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] PATH_INFO: `/v3/projects/3261fb344ed64498ad619654bd399edf/users/9c711067e61a48fd9fdbcc4d859ec459/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.098030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3261fb344ed64498ad619654bd399edf, user_id=9c711067e61a48fd9fdbcc4d859ec459, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.100926 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.114448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1147/2305] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:04 2022] PUT /identity/v3/projects/3261fb344ed64498ad619654bd399edf/users/9c711067e61a48fd9fdbcc4d859ec459/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:04.126138 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.126419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.126690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.154480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] MFA Rules not processed for user `9c711067e61a48fd9fdbcc4d859ec459`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:04.155687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:04.217149 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1159/2306] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:04 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:04.221528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.283786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df7d7f3-4ab3-4c26-8c48-fd91f7b99e08 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.288487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.289122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.289400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.289669 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.290309 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.292218 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.322409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1148/2307] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:04 2022] POST /identity/v3/projects => generated 361 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:04.326951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.337431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dbaff9-80ca-4eb4-bc39-9f921962f68d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.392372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.393010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.393284 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.393547 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.394169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.396054 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.417840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1160/2308] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:04 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:04.421962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.432522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acb4d82-a989-49f0-bb05-5abe6ba22361 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.437041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.437618 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.437891 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.438154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.438781 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.440376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.450958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1149/2309] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:04.455019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.464796 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d288607a-6d5a-42c8-bb22-f12abd047fd0 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.469431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.469989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.470289 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.470558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] PATH_INFO: `/v3/projects/f1d7616a3c7b4ea988bfa76e6bd30aa7/users/da3bc03dfac242709124bbb9af97872b/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.485683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1d7616a3c7b4ea988bfa76e6bd30aa7, user_id=da3bc03dfac242709124bbb9af97872b, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.488319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.501388 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1161/2310] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:04 2022] PUT /identity/v3/projects/f1d7616a3c7b4ea988bfa76e6bd30aa7/users/da3bc03dfac242709124bbb9af97872b/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:04.505188 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.514958 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a52c30-43c5-4691-bcef-78a1ee263053 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.571240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c04f259-ba46-4dc5-b595-aa492a264b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.571839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04f259-ba46-4dc5-b595-aa492a264b73 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.572141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04f259-ba46-4dc5-b595-aa492a264b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.572407 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04f259-ba46-4dc5-b595-aa492a264b73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.573089 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c04f259-ba46-4dc5-b595-aa492a264b73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.574687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c04f259-ba46-4dc5-b595-aa492a264b73 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.584107 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1150/2311] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:04 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:04.588295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.593786 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:04.599755 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c7505-c8e5-4096-b390-7352838132c3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.605496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.606133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.606403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.606667 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] PATH_INFO: `/v3/projects/f1d7616a3c7b4ea988bfa76e6bd30aa7/users/da3bc03dfac242709124bbb9af97872b/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.608567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1d7616a3c7b4ea988bfa76e6bd30aa7, user_id=da3bc03dfac242709124bbb9af97872b, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.611259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.621960 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1162/2312] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:04 2022] PUT /identity/v3/projects/f1d7616a3c7b4ea988bfa76e6bd30aa7/users/da3bc03dfac242709124bbb9af97872b/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:04.632394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.632673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.632967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.657755 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:04.661387 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] MFA Rules not processed for user `da3bc03dfac242709124bbb9af97872b`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:04.663244 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:04.730908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1151/2313] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:04 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:04.742164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.743840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:04.800358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564b54b-7792-48ab-858f-fecbd5092049 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.803969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.804559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.804866 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.805135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.813950 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.815349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.825889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1163/2314] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:04 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:04.899155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:04.909336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1c66f-7fe8-42cc-9ad5-5c07d11dcd1a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:04.912901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:04.913490 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:04.913763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:04.914031 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:04.969848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:04.971290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:04.981209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1152/2315] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:04 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:07.367719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:07.369169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:07.424613 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42352b1c-ae04-4a4e-8d5a-35b851c37969 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:07.428326 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:07.428964 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:07.429242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:07.429514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:07.431196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:07.492677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:07.494110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:07.503659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1164/2316] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:10.459602 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:10.461119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:10.476141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cdd8d7-cb76-4954-baef-49370e58c367 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:10.481792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:10.482555 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:10.482896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:10.483315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] PATH_INFO: `/v3/users/43f16ca13925411fb76a330b6c95395d` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:10.484514 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] RBAC: Authorizing `identity:delete_user(user_id=43f16ca13925411fb76a330b6c95395d)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:10.486842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:10.538399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43f16ca13925411fb76a330b6c95395d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.552408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43f16ca13925411fb76a330b6c95395d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.553183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43f16ca13925411fb76a330b6c95395d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.562678 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43f16ca13925411fb76a330b6c95395d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.563499 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d685376f-fa2a-43cf-9236-b2dfe6b4452c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43f16ca13925411fb76a330b6c95395d'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.577869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1153/2317] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:10 2022] DELETE /identity/v3/users/43f16ca13925411fb76a330b6c95395d => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:10.753528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:10.764037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624455c3-e731-4678-b71e-e21bea5f5937 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:10.804271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:10.821660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:10.822366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:10.822660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:10.822966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] PATH_INFO: `/v3/projects/7f6eb3e59c464c749edb69e8370f54e5` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:10.824212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f6eb3e59c464c749edb69e8370f54e5)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:10.826332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:10.872769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f6eb3e59c464c749edb69e8370f54e5', 'request_id': 'req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.878200 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f6eb3e59c464c749edb69e8370f54e5', 'request_id': 'req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.885303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f6eb3e59c464c749edb69e8370f54e5', 'request_id': 'req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.896279 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] The token cache is being invalidate because project 7f6eb3e59c464c749edb69e8370f54e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:10.896628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.897955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-124d1b5a-e9e3-47e9-96e6-0bc2689e8809 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:10.904731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1165/2318] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:10 2022] DELETE /identity/v3/projects/7f6eb3e59c464c749edb69e8370f54e5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:10.906073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:10.906361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:10.906660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:10.926843 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:10.928056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:10.932293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:10.988667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1154/2319] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:37:10.996562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:10.996886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:10.997159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.016877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:11.018003 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:11.026660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1166/2320] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:11.030917 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.040517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea940a0-98cc-4a13-b0d1-09e1ea42d6a4 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.045539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.046135 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.046448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.046714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.047397 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:11.047695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.049416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.062495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1155/2321] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:11.065689 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.075893 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b71d799-8b69-49a4-8ba8-29ed5e2342b3 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.080651 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.081274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.081549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.081819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.082445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.084341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.114156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1167/2322] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:11 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:11.115262 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.126010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab22ea3a-0184-4223-be69-a34165301e96 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.183743 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.184360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.184640 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.184939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.185574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.187518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.209356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1156/2323] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:11 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:37:11.213816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.224799 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ecb6b-ce93-4ee1-b629-21a628ff8458 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.230169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.230756 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.231025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.231285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.231908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.233514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.244881 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1168/2324] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:11.249074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.259744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f09618-4de7-4d0b-95b7-7d1ab954fd74 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.264448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca2262-825f-4826-8885-5451bc7e4e5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.265096 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca2262-825f-4826-8885-5451bc7e4e5b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.265374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca2262-825f-4826-8885-5451bc7e4e5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.265644 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca2262-825f-4826-8885-5451bc7e4e5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.266282 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ca2262-825f-4826-8885-5451bc7e4e5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.267938 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ca2262-825f-4826-8885-5451bc7e4e5b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.279324 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1157/2325] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:11.283549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.293343 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6772f5ae-373e-4c3e-8ca9-5f69776efb4e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.297837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.298480 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.298736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.299000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] PATH_INFO: `/v3/projects/297b4af1911c49f3b7b01c35236f4dae/users/ea7aceaca15a47f89c674c237b0cacae/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.313104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] RBAC: Authorizing `identity:create_grant(project_id=297b4af1911c49f3b7b01c35236f4dae, user_id=ea7aceaca15a47f89c674c237b0cacae, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.315733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.329745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1169/2326] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:11 2022] PUT /identity/v3/projects/297b4af1911c49f3b7b01c35236f4dae/users/ea7aceaca15a47f89c674c237b0cacae/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:37:11.339022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.339302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.339596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.359978 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:11.361237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:11.431456 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1158/2327] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:11.441842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:11.443835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:11.505798 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d770b-57b6-479f-9e82-43e45b90a556 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:11.510060 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d49e97d-f4e8-4521-87a8-fa814f477276 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:11.510700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d49e97d-f4e8-4521-87a8-fa814f477276 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:11.510973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d49e97d-f4e8-4521-87a8-fa814f477276 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:11.511242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d49e97d-f4e8-4521-87a8-fa814f477276 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:11.520332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d49e97d-f4e8-4521-87a8-fa814f477276 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:11.521795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d49e97d-f4e8-4521-87a8-fa814f477276 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:11.531490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1170/2328] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:37:14.494713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:14.496329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:14.508310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50266a-a7a4-41ea-a5f4-3e2d18751a31 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:14.514190 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:14.515673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:14.515852 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:14.516159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] PATH_INFO: `/v3/users/9c711067e61a48fd9fdbcc4d859ec459` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:14.517288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c711067e61a48fd9fdbcc4d859ec459)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:14.519615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:14.533160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:14.609925 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c711067e61a48fd9fdbcc4d859ec459'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:14.637293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c711067e61a48fd9fdbcc4d859ec459'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:14.638300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c711067e61a48fd9fdbcc4d859ec459'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:14.653362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c711067e61a48fd9fdbcc4d859ec459'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:14.654487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c711067e61a48fd9fdbcc4d859ec459'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:14.682482 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1159/2329] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:14 2022] DELETE /identity/v3/users/9c711067e61a48fd9fdbcc4d859ec459 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:14.684185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:14.684647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:14.684770 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:14.712429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:14.714171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:14.720162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:14.801260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1171/2330] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:14.812257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:14.814477 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:14.902489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2702e-bb89-4527-a32c-d950681135e6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:14.908248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:14.908799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:14.909077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:14.909340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:14.921532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:14.922987 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:14.934436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1160/2331] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:37:15.373877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:15.375359 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:15.388132 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99a34e1-1f23-4d15-b440-88dfa9b1d52f None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:15.394659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:15.395676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:15.396207 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:15.396688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] PATH_INFO: `/v3/projects/3261fb344ed64498ad619654bd399edf` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:15.398496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3261fb344ed64498ad619654bd399edf)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:15.401217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:15.459926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3261fb344ed64498ad619654bd399edf', 'request_id': 'req-80c59fb3-bc94-4885-97e1-b8c63b9252e5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.467014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3261fb344ed64498ad619654bd399edf', 'request_id': 'req-80c59fb3-bc94-4885-97e1-b8c63b9252e5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.487270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3261fb344ed64498ad619654bd399edf', 'request_id': 'req-80c59fb3-bc94-4885-97e1-b8c63b9252e5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.505955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] The token cache is being invalidate because project 3261fb344ed64498ad619654bd399edf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:15.507431 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.508267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-80c59fb3-bc94-4885-97e1-b8c63b9252e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.515173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1172/2332] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:15 2022] DELETE /identity/v3/projects/3261fb344ed64498ad619654bd399edf => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:15.516658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:15.530956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f43ab0-9579-4dff-bb03-56e8bac09f2a service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:15.621706 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:15.622418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:15.622674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:15.622948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] PATH_INFO: `/v3/users/da3bc03dfac242709124bbb9af97872b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:15.624076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=da3bc03dfac242709124bbb9af97872b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:15.626153 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:15.678624 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:15.699965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da3bc03dfac242709124bbb9af97872b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.727929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da3bc03dfac242709124bbb9af97872b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.729305 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da3bc03dfac242709124bbb9af97872b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.740611 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da3bc03dfac242709124bbb9af97872b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.741537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da3bc03dfac242709124bbb9af97872b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:15.765276 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1161/2333] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:15 2022] DELETE /identity/v3/users/da3bc03dfac242709124bbb9af97872b => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:15.916091 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:15.916612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:15.916612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:15.967092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] MFA Rules not processed for user `ea7aceaca15a47f89c674c237b0cacae`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:15.968162 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:16.067190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1173/2334] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:16.083561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.085140 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:16.155945 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.157446 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:16.199044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:16.218761 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93621a2-1eaf-461b-b101-23a095d40f6f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.225811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f3f8b1-829b-4c6c-8375-7d83d3ea1161 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.232753 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f3f8b1-829b-4c6c-8375-7d83d3ea1161 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.233049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f3f8b1-829b-4c6c-8375-7d83d3ea1161 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.233384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f3f8b1-829b-4c6c-8375-7d83d3ea1161 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.246049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f3f8b1-829b-4c6c-8375-7d83d3ea1161 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.247615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f3f8b1-829b-4c6c-8375-7d83d3ea1161 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.255017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7bfee2-dff8-4423-9624-b9e38074d70e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.260530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1162/2335] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:16 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:16.270076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.271615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.272963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.273778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] PATH_INFO: `/v3/projects/f1d7616a3c7b4ea988bfa76e6bd30aa7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.279090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1d7616a3c7b4ea988bfa76e6bd30aa7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.283379 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.352516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1d7616a3c7b4ea988bfa76e6bd30aa7', 'request_id': 'req-aef71aef-d8c5-462d-b380-d4180967e015'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:16.359851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1d7616a3c7b4ea988bfa76e6bd30aa7', 'request_id': 'req-aef71aef-d8c5-462d-b380-d4180967e015'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:16.366976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1d7616a3c7b4ea988bfa76e6bd30aa7', 'request_id': 'req-aef71aef-d8c5-462d-b380-d4180967e015'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:16.380177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] The token cache is being invalidate because project f1d7616a3c7b4ea988bfa76e6bd30aa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:16.380477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:16.381269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-aef71aef-d8c5-462d-b380-d4180967e015 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:16.383076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1174/2336] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:16 2022] DELETE /identity/v3/projects/f1d7616a3c7b4ea988bfa76e6bd30aa7 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:37:16.389201 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.389595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.390260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.418238 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:16.419541 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:16.491494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1163/2337] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:16.501095 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14543491-566f-43a4-a293-2fdc14133767 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.501720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14543491-566f-43a4-a293-2fdc14133767 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.502169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14543491-566f-43a4-a293-2fdc14133767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.528660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-14543491-566f-43a4-a293-2fdc14133767 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:16.530199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-14543491-566f-43a4-a293-2fdc14133767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:16.547181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1175/2338] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:16.551556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.562285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7b0d46-c419-4a53-82a4-8aa55ca6d285 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.568581 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.569676 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.570224 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.570590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.571902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:16.572391 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.574789 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.591292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1164/2339] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:16.596185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14543491-566f-43a4-a293-2fdc14133767 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.606129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14543491-566f-43a4-a293-2fdc14133767 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.611137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.611759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.612038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.612308 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.613046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.614943 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.650807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1176/2340] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:16 2022] POST /identity/v3/projects => generated 363 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:16.655173 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.668956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73609a28-fc4a-4991-b775-999dd7ad8a73 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.691921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:16.740084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.740084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.740084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.740084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.740546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.745177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.782530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1165/2341] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:16 2022] POST /identity/v3/users => generated 282 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:16.784636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.802045 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6712b-caee-4f90-8ad3-e86460bc048b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.809361 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.809999 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.810274 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.810545 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.811197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.812918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.828564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1177/2342] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:37:16.833425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.850382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bde92f-53dc-484f-ab9d-1f2084328466 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.855842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da00fa-e52f-44e8-b790-2be28eb49b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.856533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da00fa-e52f-44e8-b790-2be28eb49b48 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.856902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da00fa-e52f-44e8-b790-2be28eb49b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.857239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da00fa-e52f-44e8-b790-2be28eb49b48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.857950 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0da00fa-e52f-44e8-b790-2be28eb49b48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.859739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0da00fa-e52f-44e8-b790-2be28eb49b48 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.873480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1166/2343] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:16.877649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:16.890148 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005ec25-4cb7-4100-a97c-5aab235a820d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:16.899076 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:16.900160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.901650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.902064 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] PATH_INFO: `/v3/projects/66e449e69eb04a8ba9a1b63360fa9f22/users/b40fe50003744d3bb891998df3c9d14a/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.919473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] RBAC: Authorizing `identity:create_grant(project_id=66e449e69eb04a8ba9a1b63360fa9f22, user_id=b40fe50003744d3bb891998df3c9d14a, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:16.922382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:16.939112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1178/2344] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:16 2022] PUT /identity/v3/projects/66e449e69eb04a8ba9a1b63360fa9f22/users/b40fe50003744d3bb891998df3c9d14a/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:16.951609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:16.952113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:16.952575 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:16.985301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] MFA Rules not processed for user `b40fe50003744d3bb891998df3c9d14a`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:16.986528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:17.066117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1167/2345] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:16 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:17.070849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.142733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:17.157879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18255124-a673-4d0a-9b08-e26e8df3328e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.164503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.165715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.165992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.166263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.166901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.168831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.203823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1179/2346] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:17 2022] POST /identity/v3/projects => generated 363 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:37:17.207901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.219137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735bbfc2-3385-4680-ae19-e3a91b55655a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.284277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.285240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.285646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.285905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.286754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.289078 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.319124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1168/2347] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:17 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:17.323077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.334902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e299d3ca-dfb8-49c8-88cb-538a6a0da3f1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.343051 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.344068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.344193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.344491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.345265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.347993 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.370341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1180/2348] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:37:17.371317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.389159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f607ee11-a2b4-47bc-b74c-5817b8018ac1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.395762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.396468 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.396778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.397050 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] PATH_INFO: `/v3/projects/b3a94cfa075141a5a416df9cd48bbe0a/users/1cd81c3712f54221bd1c87fb7176fbf5/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.416281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3a94cfa075141a5a416df9cd48bbe0a, user_id=1cd81c3712f54221bd1c87fb7176fbf5, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.420157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.423922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:17.439304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1169/2349] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:17 2022] PUT /identity/v3/projects/b3a94cfa075141a5a416df9cd48bbe0a/users/1cd81c3712f54221bd1c87fb7176fbf5/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:37:17.440870 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.451605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8252e-f58a-4614-ba57-870a91ead068 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.525087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34ebae-0932-44d2-9f54-225308bb65b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.525776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34ebae-0932-44d2-9f54-225308bb65b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.526053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34ebae-0932-44d2-9f54-225308bb65b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.526322 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34ebae-0932-44d2-9f54-225308bb65b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.526955 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb34ebae-0932-44d2-9f54-225308bb65b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.528914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb34ebae-0932-44d2-9f54-225308bb65b1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.539220 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1181/2350] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:37:17.543879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.559366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9334ddbb-f2a2-4d7d-86eb-0204d077bbce admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.564134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.564822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.565120 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.565390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] PATH_INFO: `/v3/projects/b3a94cfa075141a5a416df9cd48bbe0a/users/1cd81c3712f54221bd1c87fb7176fbf5/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.567430 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3a94cfa075141a5a416df9cd48bbe0a, user_id=1cd81c3712f54221bd1c87fb7176fbf5, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.570300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.581425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1170/2351] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:17 2022] PUT /identity/v3/projects/b3a94cfa075141a5a416df9cd48bbe0a/users/1cd81c3712f54221bd1c87fb7176fbf5/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:17.592578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.592930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.593211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.625980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] MFA Rules not processed for user `1cd81c3712f54221bd1c87fb7176fbf5`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:17.627184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:17.708617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1182/2352] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:17 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:17.720936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.722795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:17.789288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b69ecd-0997-4184-b768-cf808fea13bb admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.793528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05179f5-cc4b-45f7-b09b-71c1c127aa70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.794146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05179f5-cc4b-45f7-b09b-71c1c127aa70 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.794424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05179f5-cc4b-45f7-b09b-71c1c127aa70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.794695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05179f5-cc4b-45f7-b09b-71c1c127aa70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.804818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05179f5-cc4b-45f7-b09b-71c1c127aa70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:17.806408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05179f5-cc4b-45f7-b09b-71c1c127aa70 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:17.821792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1171/2353] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:17 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:17.906413 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:17.918118 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bb370d-b1dc-4888-a478-567e2b373a81 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:17.922249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:17.922968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:17.923183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:17.923452 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:17.986469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:18.001880 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:18.003867 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:18.013185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1183/2354] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:17 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:18.056272 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eeaf79-c7aa-4c50-a294-b37ee28e65f0 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:18.056549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eeaf79-c7aa-4c50-a294-b37ee28e65f0 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:18.056848 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eeaf79-c7aa-4c50-a294-b37ee28e65f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:18.077866 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f1eeaf79-c7aa-4c50-a294-b37ee28e65f0 None None] MFA Rules not processed for user `ea7aceaca15a47f89c674c237b0cacae`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:18.079158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f1eeaf79-c7aa-4c50-a294-b37ee28e65f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:18.151317 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1172/2355] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:20.939305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:21.069153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc27b9d-31da-4401-9e14-48694decb95a service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:21.086187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:21.087951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:21.088225 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:21.088496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] PATH_INFO: `/v3/users/b40fe50003744d3bb891998df3c9d14a` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:21.100364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b40fe50003744d3bb891998df3c9d14a)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:21.102720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:21.233302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b40fe50003744d3bb891998df3c9d14a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:21.261520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b40fe50003744d3bb891998df3c9d14a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:21.262432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b40fe50003744d3bb891998df3c9d14a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:21.270124 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b40fe50003744d3bb891998df3c9d14a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:21.271910 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b40fe50003744d3bb891998df3c9d14a'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:21.291484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1184/2356] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:20 2022] DELETE /identity/v3/users/b40fe50003744d3bb891998df3c9d14a => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:21.293406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:21.293849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:21.293968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:21.328230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:21.330121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:21.400155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1173/2357] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:21.411716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:21.413219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:21.471647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e6f39-08c3-46bb-be01-198a29e7e77d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:21.475028 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:21.475749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:21.476021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:21.476288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:21.486387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:21.487844 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:21.497982 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1185/2358] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:21.935431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:21.941903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:21.958651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305944d-dfa6-417a-bc0a-0f19901e7d65 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:21.963784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:21.964614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:21.966904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:21.966904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] PATH_INFO: `/v3/projects/66e449e69eb04a8ba9a1b63360fa9f22` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:21.968719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66e449e69eb04a8ba9a1b63360fa9f22)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:21.983501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:22.022914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:22.044410 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66e449e69eb04a8ba9a1b63360fa9f22', 'request_id': 'req-83e80d3a-c87c-49f4-85c4-8f684eb92db2'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.050090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66e449e69eb04a8ba9a1b63360fa9f22', 'request_id': 'req-83e80d3a-c87c-49f4-85c4-8f684eb92db2'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.056605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66e449e69eb04a8ba9a1b63360fa9f22', 'request_id': 'req-83e80d3a-c87c-49f4-85c4-8f684eb92db2'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.069051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] The token cache is being invalidate because project 66e449e69eb04a8ba9a1b63360fa9f22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:22.069339 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.069973 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.071577 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1174/2359] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:21 2022] DELETE /identity/v3/projects/66e449e69eb04a8ba9a1b63360fa9f22 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:22.075665 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:22.086010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd9d565-d3ea-4b60-9366-20cec0483151 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:22.090708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:22.153348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:22.154084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:22.154364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:22.154646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] PATH_INFO: `/v3/users/1cd81c3712f54221bd1c87fb7176fbf5` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:22.155811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1cd81c3712f54221bd1c87fb7176fbf5)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:22.157840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:22.206832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1cd81c3712f54221bd1c87fb7176fbf5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.225935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1cd81c3712f54221bd1c87fb7176fbf5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.226713 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1cd81c3712f54221bd1c87fb7176fbf5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.235974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1cd81c3712f54221bd1c87fb7176fbf5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.236581 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-b8ac2067-591d-4027-bafe-777212581745 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1cd81c3712f54221bd1c87fb7176fbf5'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.249659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1186/2360] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:22 2022] DELETE /identity/v3/users/1cd81c3712f54221bd1c87fb7176fbf5 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:37:22.572172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:22.573797 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:22.593030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e80d3a-c87c-49f4-85c4-8f684eb92db2 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:22.679208 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:22.680061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:22.680996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:22.681528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] PATH_INFO: `/v3/projects/b3a94cfa075141a5a416df9cd48bbe0a` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:22.682960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3a94cfa075141a5a416df9cd48bbe0a)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:22.685340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:22.736233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3a94cfa075141a5a416df9cd48bbe0a', 'request_id': 'req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.741784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3a94cfa075141a5a416df9cd48bbe0a', 'request_id': 'req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.748820 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3a94cfa075141a5a416df9cd48bbe0a', 'request_id': 'req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.767344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] The token cache is being invalidate because project b3a94cfa075141a5a416df9cd48bbe0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:22.767661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.768279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e8a5c3ae-b32c-49e4-9f56-222bbb5e11a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:22.769880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1175/2361] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:22 2022] DELETE /identity/v3/projects/b3a94cfa075141a5a416df9cd48bbe0a => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:22.774719 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:22.775004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:22.775280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:22.794940 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:22.796144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:22.858603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1187/2362] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:22.921860 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:22.922139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:22.922409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:22.982439 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:23.007319 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:23.008452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:23.026289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1176/2363] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:23.030569 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.040673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3432f63e-04b2-45e6-8096-5290e7258935 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.045157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.045742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.046013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.046276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.046901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:23.047194 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.048781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.060517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1188/2364] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:23.064610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.106982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26338c0d-2b62-4a42-9822-a0499bb23f41 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.111600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.112209 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.112487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.112780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.113415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.115326 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.142673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1177/2365] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:23 2022] POST /identity/v3/projects => generated 379 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:23.146918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.162276 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:23.192995 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0196b2-5551-4e56-8f76-2dfddfd23ca1 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.247772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.248869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.249156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.249428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.250088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.252005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.273820 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1189/2366] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:23 2022] POST /identity/v3/users => generated 290 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:37:23.278141 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.288986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368bfa7f-e607-4f20-a94c-adf285514dd9 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.294992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.295634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.295919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.296246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.297384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.298524 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.311010 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1178/2367] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:23.315111 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.325172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51a2d32-620c-4441-8ef5-86a87759e463 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.331460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1604a35f-0dfb-434c-b2cc-b39fcc3303b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.332187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1604a35f-0dfb-434c-b2cc-b39fcc3303b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.332528 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1604a35f-0dfb-434c-b2cc-b39fcc3303b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.332890 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1604a35f-0dfb-434c-b2cc-b39fcc3303b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.333595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1604a35f-0dfb-434c-b2cc-b39fcc3303b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.335417 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1604a35f-0dfb-434c-b2cc-b39fcc3303b6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.345580 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1190/2368] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:23.353689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.365343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7f155f-27cf-4d42-924c-00de95d60255 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.370971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.371612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.371885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.372147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] PATH_INFO: `/v3/projects/455cab737eb941d9a742bc765c24edae/users/c460be1135954eaabd41ff2ab7aaa8dc/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.389444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=455cab737eb941d9a742bc765c24edae, user_id=c460be1135954eaabd41ff2ab7aaa8dc, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.392100 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.405037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1179/2369] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:23 2022] PUT /identity/v3/projects/455cab737eb941d9a742bc765c24edae/users/c460be1135954eaabd41ff2ab7aaa8dc/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:23.415899 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.416262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.416605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.446578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] MFA Rules not processed for user `c460be1135954eaabd41ff2ab7aaa8dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:23.447790 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:23.516063 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1191/2370] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:23.520520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.590601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa66723c-ab79-4746-bac0-bb9666f2fbc5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.595492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.596211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.596563 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.596998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.597772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.599724 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.630529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1180/2371] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:23 2022] POST /identity/v3/projects => generated 381 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:23.635158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.646391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022651b9-52d0-4752-a633-e33547c20db0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.697199 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:23.715116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.715750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.716031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.716347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.717020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.718985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.749271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1192/2372] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:23 2022] POST /identity/v3/users => generated 297 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:23.754160 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.766641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e055c6a1-936e-41de-be27-3b3633762b76 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.771845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.772522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.772825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.773093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.773823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.775500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.787323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1181/2373] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:23.791584 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.801935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d6b4c-300c-4096-b29a-c446737e194f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.807459 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.808226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.808601 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.809007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] PATH_INFO: `/v3/projects/cb9338a1d4ea497da92a0bcdf8a62106/users/2b11f531e4074552864f13ab800c1f92/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.826419 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb9338a1d4ea497da92a0bcdf8a62106, user_id=2b11f531e4074552864f13ab800c1f92, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.829145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.842272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1193/2374] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:23 2022] PUT /identity/v3/projects/cb9338a1d4ea497da92a0bcdf8a62106/users/2b11f531e4074552864f13ab800c1f92/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:23.846371 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.857300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f76f9-6af7-403e-87df-313338262bd5 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.885885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:23.937661 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae241d7-041a-4c1d-b74f-61ca6497f15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.938380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae241d7-041a-4c1d-b74f-61ca6497f15d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.939337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae241d7-041a-4c1d-b74f-61ca6497f15d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.939337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae241d7-041a-4c1d-b74f-61ca6497f15d admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.939952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae241d7-041a-4c1d-b74f-61ca6497f15d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.941689 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae241d7-041a-4c1d-b74f-61ca6497f15d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.956574 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1182/2375] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:37:23.959589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:23.971042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6021c48-afe0-48d7-a9d4-da8b145d354c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:23.975775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:23.976436 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:23.976739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:23.977018 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] PATH_INFO: `/v3/projects/cb9338a1d4ea497da92a0bcdf8a62106/users/2b11f531e4074552864f13ab800c1f92/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:23.979961 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb9338a1d4ea497da92a0bcdf8a62106, user_id=2b11f531e4074552864f13ab800c1f92, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:23.982909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:23.994730 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1194/2376] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:23 2022] PUT /identity/v3/projects/cb9338a1d4ea497da92a0bcdf8a62106/users/2b11f531e4074552864f13ab800c1f92/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:24.005792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:24.009079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:24.009698 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:24.054778 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] MFA Rules not processed for user `2b11f531e4074552864f13ab800c1f92`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:24.056583 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:24.131983 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1183/2377] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:24 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:24.144339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:24.146128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:24.219853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c270fb55-dc99-4fd0-bfa0-8af826938f7b admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:24.223611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:24.224214 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:24.224487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:24.224787 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:24.234381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:24.235794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:24.245819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1195/2378] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:24 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:37:24.563749 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:24.573037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e691eb9-88d2-4f43-a23f-d29e602262ee None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:24.577564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2b11f531e4074552864f13ab800c1f92', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb9338a1d4ea497da92a0bcdf8a62106', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:24.578158 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:24.578437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:24.578709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:24.579359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:24.579659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:24.581310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:24.593840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1184/2379] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:24.598106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:24.607924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8fa4f-075f-400c-8921-e8b27344bc12 service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:24.612532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2b11f531e4074552864f13ab800c1f92', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb9338a1d4ea497da92a0bcdf8a62106', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:24.613157 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:24.613445 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:24.613720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:24.614345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:24.616246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:24.643358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1196/2380] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:24 2022] POST /identity/v3/projects => generated 354 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:24.647349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:24.657666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4abad-2c8f-4b4d-b1be-0d701ca36380 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:24.713180 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2b11f531e4074552864f13ab800c1f92', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb9338a1d4ea497da92a0bcdf8a62106', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:24.713777 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:24.714058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:24.714329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:24.714961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:24.715263 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:24.716901 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:24.729790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1185/2381] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:24.766764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:24.777801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ca439a-24b6-4266-8b63-8a50db58dd1a tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:24.782690 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2b11f531e4074552864f13ab800c1f92', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb9338a1d4ea497da92a0bcdf8a62106', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:24.783837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:24.784123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:24.784390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] PATH_INFO: `/v3/projects/9b4e4b5c86054a979fccd9a80604d181` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:24.785550 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorizing `identity:delete_project(project_id=9b4e4b5c86054a979fccd9a80604d181)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:24.787427 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:24.834405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b4e4b5c86054a979fccd9a80604d181', 'request_id': 'req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:24.840378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b4e4b5c86054a979fccd9a80604d181', 'request_id': 'req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:24.842850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=87,keystone:INSERT=7,keystone:DELETE=6 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:24.847779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b4e4b5c86054a979fccd9a80604d181', 'request_id': 'req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:24.860794 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] The token cache is being invalidate because project 9b4e4b5c86054a979fccd9a80604d181 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:24.861078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:24.861741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-cb1a88c3-e9d8-4f4c-8d0a-0742c0025ce8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:24.863249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1197/2382] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:24 2022] DELETE /identity/v3/projects/9b4e4b5c86054a979fccd9a80604d181 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:25.047562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:25.049294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:25.077877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=100 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:25.121772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f0139-e2f2-4992-8e51-d5b6f8128ba8 tempest-VolumeQuotaClassesTest-1127099523 tempest-VolumeQuotaClassesTest-1127099523-project-admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:25.179561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:25.180207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:25.181049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:25.181370 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] PATH_INFO: `/v3/users/c460be1135954eaabd41ff2ab7aaa8dc` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:25.182470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c460be1135954eaabd41ff2ab7aaa8dc)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:25.184399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:25.230353 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c460be1135954eaabd41ff2ab7aaa8dc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.242357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c460be1135954eaabd41ff2ab7aaa8dc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.243549 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c460be1135954eaabd41ff2ab7aaa8dc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.250921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c460be1135954eaabd41ff2ab7aaa8dc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.251573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c460be1135954eaabd41ff2ab7aaa8dc'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.264345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1186/2383] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:25 2022] DELETE /identity/v3/users/c460be1135954eaabd41ff2ab7aaa8dc => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:25.268842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:25.269342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:25.269442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:25.288052 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:25.289290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:25.346578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1198/2384] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:37:25.356500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:25.357911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:25.414162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d187c09-ae10-49e8-862f-f050c24fe7b1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:25.418274 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:25.418865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:25.419136 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:25.419400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:25.425146 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:25.429592 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:25.431707 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:25.442390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1187/2385] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:25.699696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:25.701166 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:25.715382 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a2c55-ffc9-469f-8aa4-12c27e3fe090 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:25.721433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:25.721956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:25.722228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:25.722558 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] PATH_INFO: `/v3/projects/455cab737eb941d9a742bc765c24edae` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:25.723672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=455cab737eb941d9a742bc765c24edae)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:25.725658 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:25.772232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '455cab737eb941d9a742bc765c24edae', 'request_id': 'req-53726664-1bfc-4583-accf-faf5177889d2'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.778615 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '455cab737eb941d9a742bc765c24edae', 'request_id': 'req-53726664-1bfc-4583-accf-faf5177889d2'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.785981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '455cab737eb941d9a742bc765c24edae', 'request_id': 'req-53726664-1bfc-4583-accf-faf5177889d2'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.797814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] The token cache is being invalidate because project 455cab737eb941d9a742bc765c24edae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:25.798097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.798722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.800201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1199/2386] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:25 2022] DELETE /identity/v3/projects/455cab737eb941d9a742bc765c24edae => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:25.804175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:25.813974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aa2677-0483-498c-9102-838d7a5372a5 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:25.871629 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:25.872261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:25.872537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:25.872846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] PATH_INFO: `/v3/users/2b11f531e4074552864f13ab800c1f92` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:25.873955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b11f531e4074552864f13ab800c1f92)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:25.875867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:25.929007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b11f531e4074552864f13ab800c1f92'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.942365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b11f531e4074552864f13ab800c1f92'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.943113 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b11f531e4074552864f13ab800c1f92'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.950767 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b11f531e4074552864f13ab800c1f92'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.951447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e75c6a42-02c3-44cf-8b64-1989ddd299dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b11f531e4074552864f13ab800c1f92'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:25.964717 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1188/2387] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:25 2022] DELETE /identity/v3/users/2b11f531e4074552864f13ab800c1f92 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:26.258522 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.261184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:26.288520 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53726664-1bfc-4583-accf-faf5177889d2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.353969 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.354877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.355354 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.355780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] PATH_INFO: `/v3/projects/cb9338a1d4ea497da92a0bcdf8a62106` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.357110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb9338a1d4ea497da92a0bcdf8a62106)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.359268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.381031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:26.434241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb9338a1d4ea497da92a0bcdf8a62106', 'request_id': 'req-730a1ed2-7ef6-48dc-ae5f-78abb5594025'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:26.439694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb9338a1d4ea497da92a0bcdf8a62106', 'request_id': 'req-730a1ed2-7ef6-48dc-ae5f-78abb5594025'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:26.448303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb9338a1d4ea497da92a0bcdf8a62106', 'request_id': 'req-730a1ed2-7ef6-48dc-ae5f-78abb5594025'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:26.459617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] The token cache is being invalidate because project cb9338a1d4ea497da92a0bcdf8a62106 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:26.460057 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:26.460715 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-730a1ed2-7ef6-48dc-ae5f-78abb5594025 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:26.462871 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1200/2388] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:26 2022] DELETE /identity/v3/projects/cb9338a1d4ea497da92a0bcdf8a62106 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:26.467729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.468044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.468321 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.487868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:26.489119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:26.547084 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1189/2389] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:26.554939 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.555226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.555500 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.576511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:26.577740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:26.585809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1201/2390] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:26.589905 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.599341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fab0d34-145c-477b-bb02-0d4869f6f138 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.604024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.604889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.605162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.605427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.606058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:26.606352 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.607941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.616513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:26.621699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1190/2391] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:26.624725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.634564 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8550ad-0f7e-4e03-982e-d6b0fa9031b4 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.639219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.639809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.640086 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.640355 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.641017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.642917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.670036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1202/2392] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:26 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:37:26.674436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.685899 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2abe57c-13c6-46d4-ad78-6668d021c70b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.746941 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.747721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.747999 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.748261 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.748993 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.750912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.774259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1191/2393] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:26 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:26.778862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.790299 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c42a1-92fc-4246-a628-e0ab7262170e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.794988 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.795627 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.795906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.796178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.796862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.798498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.809190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1203/2394] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:26.813329 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.824048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd677bd-1557-4d45-b106-7f3c7b0435e6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.829192 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2324c152-5c2f-4e87-9d5b-761001b04f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.829829 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2324c152-5c2f-4e87-9d5b-761001b04f0c admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.830105 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2324c152-5c2f-4e87-9d5b-761001b04f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.830373 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2324c152-5c2f-4e87-9d5b-761001b04f0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.831013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2324c152-5c2f-4e87-9d5b-761001b04f0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.832656 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2324c152-5c2f-4e87-9d5b-761001b04f0c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.843367 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1192/2395] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:26.847424 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:26.857731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bac513-6cee-4f7f-8181-e44f57785b4c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:26.862338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:26.862953 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.863230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.863517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] PATH_INFO: `/v3/projects/bfa19ff6854e4f45ad73c0a3a7ac3217/users/72831f67ac0d402d8f435c3216ca968b/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.878767 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfa19ff6854e4f45ad73c0a3a7ac3217, user_id=72831f67ac0d402d8f435c3216ca968b, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:26.881478 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:26.897245 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1204/2396] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:26 2022] PUT /identity/v3/projects/bfa19ff6854e4f45ad73c0a3a7ac3217/users/72831f67ac0d402d8f435c3216ca968b/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:26.907741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:26.908094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:26.908381 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:26.938426 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] MFA Rules not processed for user `72831f67ac0d402d8f435c3216ca968b`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:26.939617 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.004874 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1193/2397] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:37:27.015825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:27.017562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.075467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccf6d08-47d5-4606-9314-cf3fa43276c2 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:27.079303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cbadc6-1327-46b0-ae3b-efa0c6f8d05d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:27.079902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cbadc6-1327-46b0-ae3b-efa0c6f8d05d service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.080171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cbadc6-1327-46b0-ae3b-efa0c6f8d05d service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.080735 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cbadc6-1327-46b0-ae3b-efa0c6f8d05d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.089728 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cbadc6-1327-46b0-ae3b-efa0c6f8d05d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:27.091406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cbadc6-1327-46b0-ae3b-efa0c6f8d05d service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:27.101585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1205/2398] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:27 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:27.123754 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:27.190576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbb56a-ce71-4702-8776-61a4b5534b87 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:27.195359 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:27.195992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.196302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.196576 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] PATH_INFO: `/v3/users/72831f67ac0d402d8f435c3216ca968b` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.197763 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72831f67ac0d402d8f435c3216ca968b)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:27.199716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:27.211249 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:27.248721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72831f67ac0d402d8f435c3216ca968b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.265951 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72831f67ac0d402d8f435c3216ca968b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.266614 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72831f67ac0d402d8f435c3216ca968b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.273776 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72831f67ac0d402d8f435c3216ca968b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.274953 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72831f67ac0d402d8f435c3216ca968b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.288844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1194/2399] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:27 2022] DELETE /identity/v3/users/72831f67ac0d402d8f435c3216ca968b => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:27.293913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.294432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.294432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.313532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:27.314743 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.354309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:27.376716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1206/2400] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:27.386458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:27.388122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.449157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e61f08-c9cf-4ae1-93a2-bccae587c4a2 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:27.453137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0fa17c-5ab5-44a7-8607-ef81112ad260 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:27.454159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0fa17c-5ab5-44a7-8607-ef81112ad260 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.454758 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0fa17c-5ab5-44a7-8607-ef81112ad260 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.455040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0fa17c-5ab5-44a7-8607-ef81112ad260 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.465834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0fa17c-5ab5-44a7-8607-ef81112ad260 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:27.467990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0fa17c-5ab5-44a7-8607-ef81112ad260 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:27.478351 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1195/2401] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:27.747646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:27.749195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.762000 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028aa6b7-3035-4ca9-91c7-f80ff5876c62 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:27.766574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:27.767234 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.767509 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.767782 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] PATH_INFO: `/v3/projects/bfa19ff6854e4f45ad73c0a3a7ac3217` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.768958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfa19ff6854e4f45ad73c0a3a7ac3217)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:27.770922 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:27.822425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfa19ff6854e4f45ad73c0a3a7ac3217', 'request_id': 'req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.829099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfa19ff6854e4f45ad73c0a3a7ac3217', 'request_id': 'req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.837557 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfa19ff6854e4f45ad73c0a3a7ac3217', 'request_id': 'req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.848833 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] The token cache is being invalidate because project bfa19ff6854e4f45ad73c0a3a7ac3217 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:27.849186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.849841 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-5d3d8cb9-abfd-40fc-a24f-da544a9daeb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:27.851460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1207/2402] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:27 2022] DELETE /identity/v3/projects/bfa19ff6854e4f45ad73c0a3a7ac3217 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:27.856567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.857073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.857172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.876379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:27.877582 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.943679 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1196/2403] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:27.953156 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:27.953423 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:27.953686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:27.973989 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:27.975178 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:27.987471 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1208/2404] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:27.988730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:27.999677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c979dccd-773a-48d8-aef8-d145ca1e4324 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.005395 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.006222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.006501 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.006788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.007444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:37:28.007858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.010077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.026652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1197/2405] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:37:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:37:28.030130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:28.051701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e020ce7-4d73-4951-97f8-4fcea1c8ba86 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.057751 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.058570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.058907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.059277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.060011 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.062438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.095685 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1209/2406] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:37:28 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:28.099822 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:28.116969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224b1e92-7631-4e97-871a-9231c285b0d1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.131418 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:28.179455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.180130 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.181011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.181376 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.182086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.184092 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.206811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1198/2407] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:37:28 2022] POST /identity/v3/users => generated 288 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:37:28.210837 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:28.221586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a104f9-1de5-420e-8ca0-c660b7019f7d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.226391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.226996 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.227271 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.227535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.228161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.230067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.239977 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1210/2408] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:37:28.244443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:28.254568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aaff4c-dd08-4121-953c-9f8f155b6b26 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.259815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba0dfef-b56e-4659-8923-cfb17c5bbe49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.260408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba0dfef-b56e-4659-8923-cfb17c5bbe49 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.260684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba0dfef-b56e-4659-8923-cfb17c5bbe49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.260976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba0dfef-b56e-4659-8923-cfb17c5bbe49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.261610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba0dfef-b56e-4659-8923-cfb17c5bbe49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.263196 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba0dfef-b56e-4659-8923-cfb17c5bbe49 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.274118 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1199/2409] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:37:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:37:28.278263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:28.288901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db331d08-639c-46b9-8900-e72a2085086f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.293741 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.294606 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.295158 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.295272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] PATH_INFO: `/v3/projects/ca527473d42a4ed398937c1c0e15b034/users/531eba20cad04af4b0db35ab587beab3/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.311034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca527473d42a4ed398937c1c0e15b034, user_id=531eba20cad04af4b0db35ab587beab3, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.313765 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.327729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1211/2410] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:37:28 2022] PUT /identity/v3/projects/ca527473d42a4ed398937c1c0e15b034/users/531eba20cad04af4b0db35ab587beab3/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:37:28.338567 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.338900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.339171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.369915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] MFA Rules not processed for user `531eba20cad04af4b0db35ab587beab3`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:28.371323 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:28.449085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1200/2411] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:28.460821 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:28.462568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:28.527239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a74846-2424-40e7-b722-83b9e522f78c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:28.530997 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22956c5a-2d15-4eae-a746-5a397bc92d20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:28.531482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22956c5a-2d15-4eae-a746-5a397bc92d20 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:28.531759 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22956c5a-2d15-4eae-a746-5a397bc92d20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:28.532024 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22956c5a-2d15-4eae-a746-5a397bc92d20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:28.541080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22956c5a-2d15-4eae-a746-5a397bc92d20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:28.542502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22956c5a-2d15-4eae-a746-5a397bc92d20 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:28.551878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1212/2412] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:31.688642 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:31.750742 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9db66b6-e151-4cef-97d8-948addc01b14 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:31.755156 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:31.755762 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:31.756032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:31.756296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] PATH_INFO: `/v3/users/531eba20cad04af4b0db35ab587beab3` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:31.757402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] RBAC: Authorizing `identity:delete_user(user_id=531eba20cad04af4b0db35ab587beab3)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:31.759907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:31.788257 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:31.809921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '531eba20cad04af4b0db35ab587beab3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:31.822634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '531eba20cad04af4b0db35ab587beab3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:31.823417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '531eba20cad04af4b0db35ab587beab3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:31.830169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '531eba20cad04af4b0db35ab587beab3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:31.830869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '531eba20cad04af4b0db35ab587beab3'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:31.843015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1201/2413] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:31 2022] DELETE /identity/v3/users/531eba20cad04af4b0db35ab587beab3 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:31.847477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:31.847931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:31.848049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:31.856183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:31.869621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:31.871198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:31.934103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1213/2414] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:37:31.943653 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:31.945085 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:32.025121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fc7478-db28-4d66-9130-94321b2b859b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:32.028687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:32.029315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:32.029588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:32.029854 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:32.038613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:32.040087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:32.050804 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1202/2415] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:32.323301 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:32.324830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:32.337612 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff654bd6-7000-41ab-a767-61a197816114 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:32.342705 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:32.343420 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:32.344204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:32.344204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] PATH_INFO: `/v3/projects/ca527473d42a4ed398937c1c0e15b034` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:32.345290 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca527473d42a4ed398937c1c0e15b034)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:32.347179 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:32.398563 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca527473d42a4ed398937c1c0e15b034', 'request_id': 'req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:32.404167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca527473d42a4ed398937c1c0e15b034', 'request_id': 'req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:32.411303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca527473d42a4ed398937c1c0e15b034', 'request_id': 'req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:32.422941 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] The token cache is being invalidate because project ca527473d42a4ed398937c1c0e15b034 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:32.423228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:32.423853 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-fed9c05a-292d-4b4b-af5a-7284f7ba80ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:32.425375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1214/2416] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:32 2022] DELETE /identity/v3/projects/ca527473d42a4ed398937c1c0e15b034 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:33.043705 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:33.045128 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:33.057682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e28cf5-875c-45a3-98dc-f4f9d1d1cb39 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:33.113558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:33.114197 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:33.114474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:33.114744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] PATH_INFO: `/v3/users/b8cff3ae919549ef8cd6d18650903fbf` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:33.115882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8cff3ae919549ef8cd6d18650903fbf)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:33.117863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:33.164191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8cff3ae919549ef8cd6d18650903fbf'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.166058 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:33.178564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8cff3ae919549ef8cd6d18650903fbf'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.179260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8cff3ae919549ef8cd6d18650903fbf'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.186523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8cff3ae919549ef8cd6d18650903fbf'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.187164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8cff3ae919549ef8cd6d18650903fbf'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.200239 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1203/2417] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:37:33 2022] DELETE /identity/v3/users/b8cff3ae919549ef8cd6d18650903fbf => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:33.204836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:33.205317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:33.205428 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:33.224467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:37:33.225709 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:33.284297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1215/2418] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:37:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:37:33.294129 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:33.295518 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:33.351966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf89b2-492e-4fcc-9647-7024b2a4a0a1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:33.356604 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:33.357447 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:33.357591 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:33.357827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:33.368561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:33.370066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:33.379537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1204/2419] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:37:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:37:33.640453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:37:33.641926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:37:33.653686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027c8281-d910-4cd3-b462-055f76a06d47 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:37:33.658409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:37:33.659035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:37:33.659305 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:37:33.659568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] PATH_INFO: `/v3/projects/048b0e277eab4dcf94114c329e8263ab` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:37:33.660722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=048b0e277eab4dcf94114c329e8263ab)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:37:33.662647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:37:33.687775 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:33.711230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '048b0e277eab4dcf94114c329e8263ab', 'request_id': 'req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.716640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '048b0e277eab4dcf94114c329e8263ab', 'request_id': 'req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.724213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '048b0e277eab4dcf94114c329e8263ab', 'request_id': 'req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.736033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] The token cache is being invalidate because project 048b0e277eab4dcf94114c329e8263ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:37:33.736467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.737101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-02ae2cfc-84b9-4579-981b-9ecb88fcdf1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:37:33.738556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1216/2420] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:37:33 2022] DELETE /identity/v3/projects/048b0e277eab4dcf94114c329e8263ab => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:37:43.375405 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=37,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:37:43.733104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=15,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:38:35.725219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:38:35.726638 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:38:35.814185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918a2865-07fe-4795-ae73-ac0505d5c025 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:38:35.872008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:38:35.872598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:35.872897 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:35.873161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:35.874827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:38:35.885513 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:38:35.887074 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e899b-2a33-42ae-b299-3af7d92f721a service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:38:35.897175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1205/2421] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:38:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:38:45.893039 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=67 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:38:55.722583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12273648-dde8-49a3-8d4b-94db74a7b3f5 None None] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:55.723183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12273648-dde8-49a3-8d4b-94db74a7b3f5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:55.723183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12273648-dde8-49a3-8d4b-94db74a7b3f5 None None] PATH_INFO: `/` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:55.724464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1217/2422] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:38:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 12 02:38:55.731404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9553467-d640-4c3e-8d0e-ade794ca2275 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:55.731887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9553467-d640-4c3e-8d0e-ade794ca2275 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:55.732000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9553467-d640-4c3e-8d0e-ade794ca2275 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:55.777842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c9553467-d640-4c3e-8d0e-ade794ca2275 None None] MFA Rules not processed for user `8c5cd3df4be042f7ac338b9a6b0a50da`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:38:55.779073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c9553467-d640-4c3e-8d0e-ade794ca2275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:38:55.848248 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1206/2423] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:38:55 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:38:55.861080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12273648-dde8-49a3-8d4b-94db74a7b3f5 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:38:55.862763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-12273648-dde8-49a3-8d4b-94db74a7b3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:38:55.876249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12273648-dde8-49a3-8d4b-94db74a7b3f5 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:38:55.879725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33892b51-a73c-4a99-85ad-c0ba2a933e06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:38:55.880306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33892b51-a73c-4a99-85ad-c0ba2a933e06 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:55.880572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33892b51-a73c-4a99-85ad-c0ba2a933e06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:55.881469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33892b51-a73c-4a99-85ad-c0ba2a933e06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:55.888216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33892b51-a73c-4a99-85ad-c0ba2a933e06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:38:55.890203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33892b51-a73c-4a99-85ad-c0ba2a933e06 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:38:55.897703 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1218/2424] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:38:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:38:56.989396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e57229e-dc67-4e01-990c-ff0b18c4a8ae None None] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:56.989921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e57229e-dc67-4e01-990c-ff0b18c4a8ae None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:56.989921 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e57229e-dc67-4e01-990c-ff0b18c4a8ae None None] PATH_INFO: `/` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:56.991711 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1207/2425] 10.222.0.23 () {54 vars in 921 bytes} [Wed Oct 12 02:38:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 12 02:38:56.997114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:56.997568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:56.997669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:57.029037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] MFA Rules not processed for user `aa015e7713c34f16be4529081c849ccf`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:38:57.030754 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:38:57.118044 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1219/2426] 10.222.0.23 () {58 vars in 1018 bytes} [Wed Oct 12 02:38:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:38:57.127991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e57229e-dc67-4e01-990c-ff0b18c4a8ae None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:38:57.139043 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e57229e-dc67-4e01-990c-ff0b18c4a8ae None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:38:57.142560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:38:57.143144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:38:57.143417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:38:57.143685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:38:57.153470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:38:57.155411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:38:57.165871 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1208/2427] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:38:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:39:07.114127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=49 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:39:07.161304 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=42 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:39:45.547472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:39:45.557575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb84842-569b-4fe4-be96-f70580a79566 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:39:45.561174 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:39:45.561771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:39:45.562043 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:39:45.562337 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:39:45.564087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:39:45.577463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:39:45.578931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a6d183-f1fa-425c-86b2-0e1cf67dd6f5 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:39:45.589731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1220/2428] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:39:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:39:46.763960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:39:46.771186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:39:46.776963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f39519-60cd-48b8-925f-700f842677a7 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:39:46.780559 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:39:46.781185 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:39:46.781461 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:39:46.781745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:39:46.783974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:39:46.796996 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:39:46.798985 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:39:46.809756 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1209/2429] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:39:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:39:51.344013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:39:51.345061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:39:51.345171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:39:51.364636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] MFA Rules not processed for user `ea7aceaca15a47f89c674c237b0cacae`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:39:51.365855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:39:51.419186 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1221/2430] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:39:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:39:51.717515 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:39:51.726793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bd7dd-2ec0-464f-8456-8f764f7ebe0e service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:39:51.730372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:39:51.730954 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:39:51.731236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:39:51.731502 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:39:51.733227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:39:51.742655 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:39:51.744076 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:39:51.752172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1210/2431] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:39:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:40:01.412552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=31 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:01.748460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=12 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:28.246682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:40:28.248246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:40:28.269585 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b9ed0-3ee2-42d4-a873-ac11db5034ff None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:40:28.274285 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:40:28.274981 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:40:28.275323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:40:28.275662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:40:28.277560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:40:28.290857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:40:28.292639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:40:28.304604 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1222/2432] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:40:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:40:28.400211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:40:28.411620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a809ed93-d7bb-4acd-88fd-f4a2a79db5d9 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:40:28.415366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:40:28.416122 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:40:28.416398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:40:28.416823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:40:28.418720 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:40:28.429423 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:40:28.430865 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:40:28.439966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1211/2433] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:40:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:40:38.300116 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=6 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:38.377297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:40:38.378811 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:40:38.435832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:38.451670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b459c13-1f4c-4e31-92b9-629d531c7815 service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:40:38.456230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:40:38.457198 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:40:38.457477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:40:38.457795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] PATH_INFO: `/v3/users/60e6b5fcfd9d454bb10dd314bcdbedf6` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:40:38.458934 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=60e6b5fcfd9d454bb10dd314bcdbedf6)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:40:38.460917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:40:38.513897 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60e6b5fcfd9d454bb10dd314bcdbedf6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.525876 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60e6b5fcfd9d454bb10dd314bcdbedf6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.526570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60e6b5fcfd9d454bb10dd314bcdbedf6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.534006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60e6b5fcfd9d454bb10dd314bcdbedf6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.534684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60e6b5fcfd9d454bb10dd314bcdbedf6'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.550273 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1223/2434] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:40:38 2022] DELETE /identity/v3/users/60e6b5fcfd9d454bb10dd314bcdbedf6 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:40:38.733290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:40:38.752660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f057cee-916b-4deb-bff1-00d190314a1c service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:40:38.818798 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:40:38.819436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:40:38.819716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:40:38.819979 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] PATH_INFO: `/v3/projects/b1601a3e4f41405d8e9001af6da86b46` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:40:38.821361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1601a3e4f41405d8e9001af6da86b46)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:40:38.823365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:40:38.869336 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1601a3e4f41405d8e9001af6da86b46', 'request_id': 'req-4b27b847-ba34-48a8-98d0-1e01e96aec13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.874812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1601a3e4f41405d8e9001af6da86b46', 'request_id': 'req-4b27b847-ba34-48a8-98d0-1e01e96aec13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.880990 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1601a3e4f41405d8e9001af6da86b46', 'request_id': 'req-4b27b847-ba34-48a8-98d0-1e01e96aec13'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.892595 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] The token cache is being invalidate because project b1601a3e4f41405d8e9001af6da86b46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:40:38.892981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.893610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:38.895652 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1212/2435] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:40:38 2022] DELETE /identity/v3/projects/b1601a3e4f41405d8e9001af6da86b46 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:40:38.899561 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:40:38.900986 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:40:38.914306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c97638-6cc4-4d55-8620-23ac9c6351b4 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:40:38.969515 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:40:38.970144 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:40:38.970421 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:40:38.970694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] PATH_INFO: `/v3/users/3ea8132514d646f8865347accfc23dda` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:40:38.971823 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ea8132514d646f8865347accfc23dda)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:40:38.974620 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:40:38.995458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:39.019463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ea8132514d646f8865347accfc23dda'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.031625 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ea8132514d646f8865347accfc23dda'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.032277 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ea8132514d646f8865347accfc23dda'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.040974 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ea8132514d646f8865347accfc23dda'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.041598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ea8132514d646f8865347accfc23dda'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.055130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1224/2436] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:40:38 2022] DELETE /identity/v3/users/3ea8132514d646f8865347accfc23dda => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:40:39.228047 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:40:39.229606 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:40:39.242842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b27b847-ba34-48a8-98d0-1e01e96aec13 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:40:39.301044 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:40:39.301670 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:40:39.301944 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:40:39.302206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] PATH_INFO: `/v3/projects/e17aa6bdc3204e02a0ffae37efac63b4` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:40:39.303296 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e17aa6bdc3204e02a0ffae37efac63b4)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:40:39.305223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:40:39.339550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:39.353408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e17aa6bdc3204e02a0ffae37efac63b4', 'request_id': 'req-f0086749-14e9-4af0-9a3c-a664a42d8629'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.358932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e17aa6bdc3204e02a0ffae37efac63b4', 'request_id': 'req-f0086749-14e9-4af0-9a3c-a664a42d8629'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.367542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e17aa6bdc3204e02a0ffae37efac63b4', 'request_id': 'req-f0086749-14e9-4af0-9a3c-a664a42d8629'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.380065 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] The token cache is being invalidate because project e17aa6bdc3204e02a0ffae37efac63b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:40:39.380349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.380991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:40:39.382494 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1213/2437] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:40:39 2022] DELETE /identity/v3/projects/e17aa6bdc3204e02a0ffae37efac63b4 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:40:49.051437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=7,keystone:SELECT=20,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:40:49.376506 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:11.239915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:11.239915 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:11.309269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:11.311401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:11.344437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14014182-4cfa-4a81-bb3b-31272bc16dfd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:11.394277 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0086749-14e9-4af0-9a3c-a664a42d8629 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:11.399230 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:11.399858 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.400133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.400398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] PATH_INFO: `/v3/users/ea7aceaca15a47f89c674c237b0cacae` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.402240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea7aceaca15a47f89c674c237b0cacae)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:11.404455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:11.422451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:11.423097 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.423371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.423640 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.425387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:11.437681 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:11.439262 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:11.450215 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1225/2438] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:11.454631 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea7aceaca15a47f89c674c237b0cacae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.469626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea7aceaca15a47f89c674c237b0cacae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.469998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea7aceaca15a47f89c674c237b0cacae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.478300 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea7aceaca15a47f89c674c237b0cacae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.479079 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f26f4d61-dce8-4688-8d4c-660b54f89652 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea7aceaca15a47f89c674c237b0cacae'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.493700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1214/2439] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:11 2022] DELETE /identity/v3/users/ea7aceaca15a47f89c674c237b0cacae => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:11.663374 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:11.674135 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56528570-a555-483c-9e51-3e89a7c8d969 service placement] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:11.732358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:11.733014 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.733294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.733568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] PATH_INFO: `/v3/projects/297b4af1911c49f3b7b01c35236f4dae` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.734677 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=297b4af1911c49f3b7b01c35236f4dae)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:11.736553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:11.772809 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:11.783781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '297b4af1911c49f3b7b01c35236f4dae', 'request_id': 'req-c62b4e46-b032-4880-85fd-7b45dfc7aae9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.789828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '297b4af1911c49f3b7b01c35236f4dae', 'request_id': 'req-c62b4e46-b032-4880-85fd-7b45dfc7aae9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.796803 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '297b4af1911c49f3b7b01c35236f4dae', 'request_id': 'req-c62b4e46-b032-4880-85fd-7b45dfc7aae9'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.807772 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] The token cache is being invalidate because project 297b4af1911c49f3b7b01c35236f4dae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:11.808055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.808676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-c62b4e46-b032-4880-85fd-7b45dfc7aae9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:11.810183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1226/2440] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:11 2022] DELETE /identity/v3/projects/297b4af1911c49f3b7b01c35236f4dae => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:11.815070 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.815357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.815630 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.835510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:11.836729 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:11.895318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1215/2441] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:11.904802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c026195e-8538-4920-9c69-f3786537c942 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.905173 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c026195e-8538-4920-9c69-f3786537c942 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.905434 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c026195e-8538-4920-9c69-f3786537c942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.924924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-c026195e-8538-4920-9c69-f3786537c942 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:11.926071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-c026195e-8538-4920-9c69-f3786537c942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:11.933827 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1227/2442] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:11.937914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:11.947635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8c59e-1327-48fe-9387-48568cf97f23 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:11.952799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:11.953398 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.953666 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.953934 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.954569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:41:11.954873 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:11.956471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:11.968487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1216/2443] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:41:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:41:11.972616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c026195e-8538-4920-9c69-f3786537c942 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:11.982025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c026195e-8538-4920-9c69-f3786537c942 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:11.986607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:11.987211 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:11.987488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:11.987752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:11.988367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:11.990259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.026107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1228/2444] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:11 2022] POST /identity/v3/projects => generated 393 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:12.030431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.040171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142c524-cc80-41d7-941d-316619c00884 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.043697 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:12.098880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.099486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.099759 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.100027 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.100928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.102882 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.124948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1217/2445] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:12 2022] POST /identity/v3/users => generated 297 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:12.129050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.139967 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b07be10-8f7b-4145-912e-7e8bb7c32aea admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.145201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.145806 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.146099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.146370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.147004 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.148576 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.158426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1229/2446] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:12.162343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.172051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ac9552-825c-4cf8-abe3-4062d668506f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.176612 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d5a9d-acab-4863-a6fc-86ada7313517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.177167 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d5a9d-acab-4863-a6fc-86ada7313517 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.177442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d5a9d-acab-4863-a6fc-86ada7313517 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.177709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d5a9d-acab-4863-a6fc-86ada7313517 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.178334 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d5a9d-acab-4863-a6fc-86ada7313517 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.179919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d5a9d-acab-4863-a6fc-86ada7313517 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.191247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1218/2447] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:12.195594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.206161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9101e84-cfbd-4d86-b32d-f35604971a26 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.210723 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.211364 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.211918 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.212192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] PATH_INFO: `/v3/projects/1a9a73dd2b3144ed881765ac7cdecdad/users/65d58f28ecce4e9687a71f2c19a33892/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.226877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a9a73dd2b3144ed881765ac7cdecdad, user_id=65d58f28ecce4e9687a71f2c19a33892, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.229976 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.242335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1230/2448] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:12 2022] PUT /identity/v3/projects/1a9a73dd2b3144ed881765ac7cdecdad/users/65d58f28ecce4e9687a71f2c19a33892/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:12.253163 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.253446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.253716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.281294 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] MFA Rules not processed for user `65d58f28ecce4e9687a71f2c19a33892`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:12.282488 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:12.346162 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1219/2449] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:12.351032 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.411147 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3e67e6-5d66-42c7-a7a8-5c3756053ee3 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.415729 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.416319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.416589 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.416884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.417506 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.419440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.445746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1231/2450] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:12 2022] POST /identity/v3/projects => generated 393 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:12.450347 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.461023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44708b28-a0c6-4708-94b5-cedc4dd71fa3 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.516048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.516650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.516962 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.517229 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.517846 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.519773 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.540878 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1220/2451] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:12 2022] POST /identity/v3/users => generated 303 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:12.544905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.555127 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9b7dd2-1541-4a8d-b4f1-08c057ec4f82 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.559745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.560339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.560622 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.560932 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.561562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.563154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.574035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1232/2452] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:12.577910 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.584170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:12.588080 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b72d4-b474-43f3-8b47-9515de72f5e0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.594372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.595012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.595288 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.595628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] PATH_INFO: `/v3/projects/dbc51ed3ab6e4666b9e8085f2ceba418/users/2650c33d7c7a43f6afb9139e753f4610/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.610510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc51ed3ab6e4666b9e8085f2ceba418, user_id=2650c33d7c7a43f6afb9139e753f4610, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.613234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.625700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1221/2453] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:12 2022] PUT /identity/v3/projects/dbc51ed3ab6e4666b9e8085f2ceba418/users/2650c33d7c7a43f6afb9139e753f4610/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:12.629763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.640164 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff80be9-0ead-43c4-92ff-4405c79cb94f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.660023 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:12.702402 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6723c0ad-7754-40b1-89e8-ddf4a9d0f447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.702938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6723c0ad-7754-40b1-89e8-ddf4a9d0f447 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.703210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6723c0ad-7754-40b1-89e8-ddf4a9d0f447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.703473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6723c0ad-7754-40b1-89e8-ddf4a9d0f447 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.704096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6723c0ad-7754-40b1-89e8-ddf4a9d0f447 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.705748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6723c0ad-7754-40b1-89e8-ddf4a9d0f447 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.716444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1233/2454] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:41:12.720911 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.731341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e701c41-ee87-41cd-ba1e-be293d767c59 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.735825 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.736462 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.736857 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.737038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] PATH_INFO: `/v3/projects/dbc51ed3ab6e4666b9e8085f2ceba418/users/2650c33d7c7a43f6afb9139e753f4610/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.739004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc51ed3ab6e4666b9e8085f2ceba418, user_id=2650c33d7c7a43f6afb9139e753f4610, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.741659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.752827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1222/2455] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:12 2022] PUT /identity/v3/projects/dbc51ed3ab6e4666b9e8085f2ceba418/users/2650c33d7c7a43f6afb9139e753f4610/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:12.764104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.764453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.764878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.792911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] MFA Rules not processed for user `2650c33d7c7a43f6afb9139e753f4610`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:12.794107 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:12.863975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1234/2456] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:12 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:12.876675 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:12.878394 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:12.958008 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a61ad7-8886-4503-8498-b5cef41b48f3 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:12.961564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:12.962150 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:12.962420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:12.962685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:12.971451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:12.972955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:12.982406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1223/2457] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:12 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:13.027931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:13.029338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:13.041183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db796d2c-ad01-48c9-bd7e-ae9e340eb8bd None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:13.044749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b705e162-d915-47b8-bafc-4e8ec708ead2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:13.045341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b705e162-d915-47b8-bafc-4e8ec708ead2 service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:13.045614 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b705e162-d915-47b8-bafc-4e8ec708ead2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:13.045886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b705e162-d915-47b8-bafc-4e8ec708ead2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:13.101281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b705e162-d915-47b8-bafc-4e8ec708ead2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:13.102750 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b705e162-d915-47b8-bafc-4e8ec708ead2 service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:13.112405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1235/2458] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:16.377589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:16.438157 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55e991-6a95-495c-8fea-4d0f66423909 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:16.442588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:16.443193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:16.443463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:16.443728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] PATH_INFO: `/v3/users/65d58f28ecce4e9687a71f2c19a33892` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:16.444840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65d58f28ecce4e9687a71f2c19a33892)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:16.446747 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:16.493683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65d58f28ecce4e9687a71f2c19a33892'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:16.498133 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:16.507416 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65d58f28ecce4e9687a71f2c19a33892'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:16.508081 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65d58f28ecce4e9687a71f2c19a33892'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:16.515051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65d58f28ecce4e9687a71f2c19a33892'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:16.515671 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65d58f28ecce4e9687a71f2c19a33892'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:16.528392 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1224/2459] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:16 2022] DELETE /identity/v3/users/65d58f28ecce4e9687a71f2c19a33892 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:16.532909 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:16.533380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:16.533498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:16.552636 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:16.553858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:16.593549 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:16.613297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1236/2460] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:16.623009 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:16.624391 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:16.681330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ef15e3-eced-4069-9615-a327f14484f1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:16.684991 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:16.685588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:16.685859 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:16.686124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:16.694788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:16.696207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:16.705673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1225/2461] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:16.966672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:16.968187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:16.981366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2e261b-d858-48a3-b7fb-9b2f07f5cf36 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:16.986050 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:16.986686 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:16.986963 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:16.987228 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] PATH_INFO: `/v3/projects/1a9a73dd2b3144ed881765ac7cdecdad` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:16.988342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a9a73dd2b3144ed881765ac7cdecdad)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:16.990323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.036270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a9a73dd2b3144ed881765ac7cdecdad', 'request_id': 'req-505e8ff7-5529-4383-ba70-d810fabab76f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.041801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a9a73dd2b3144ed881765ac7cdecdad', 'request_id': 'req-505e8ff7-5529-4383-ba70-d810fabab76f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.048850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a9a73dd2b3144ed881765ac7cdecdad', 'request_id': 'req-505e8ff7-5529-4383-ba70-d810fabab76f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.059747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] The token cache is being invalidate because project 1a9a73dd2b3144ed881765ac7cdecdad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:17.060029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.060659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.062150 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1237/2462] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:16 2022] DELETE /identity/v3/projects/1a9a73dd2b3144ed881765ac7cdecdad => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:17.066132 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.076206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0b7d4-dd97-4cee-a45c-97ed69efb49d service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.131794 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.132417 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.132714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.132986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] PATH_INFO: `/v3/users/2650c33d7c7a43f6afb9139e753f4610` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.134088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2650c33d7c7a43f6afb9139e753f4610)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.136519 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.182967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2650c33d7c7a43f6afb9139e753f4610'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.195067 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2650c33d7c7a43f6afb9139e753f4610'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.195752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2650c33d7c7a43f6afb9139e753f4610'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.202902 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2650c33d7c7a43f6afb9139e753f4610'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.203657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-73e3fd27-ce94-4c83-ad4b-bdc83da64d16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2650c33d7c7a43f6afb9139e753f4610'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.205713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:17.217833 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1226/2463] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:17 2022] DELETE /identity/v3/users/2650c33d7c7a43f6afb9139e753f4610 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:17.486389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.488185 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:17.502516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e8ff7-5529-4383-ba70-d810fabab76f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.562517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.563190 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.563403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.563669 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] PATH_INFO: `/v3/projects/dbc51ed3ab6e4666b9e8085f2ceba418` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.564792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbc51ed3ab6e4666b9e8085f2ceba418)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.566699 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.612229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbc51ed3ab6e4666b9e8085f2ceba418', 'request_id': 'req-29664e14-5c1c-416b-beda-802902301974'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.617740 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbc51ed3ab6e4666b9e8085f2ceba418', 'request_id': 'req-29664e14-5c1c-416b-beda-802902301974'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.625309 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbc51ed3ab6e4666b9e8085f2ceba418', 'request_id': 'req-29664e14-5c1c-416b-beda-802902301974'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.632263 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:17.637401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] The token cache is being invalidate because project dbc51ed3ab6e4666b9e8085f2ceba418 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:17.637889 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.638919 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-29664e14-5c1c-416b-beda-802902301974 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:17.640479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1238/2464] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:17 2022] DELETE /identity/v3/projects/dbc51ed3ab6e4666b9e8085f2ceba418 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:17.645362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.645856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.645968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.665066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:17.666295 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:17.723964 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1227/2465] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:17.732498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.732814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.733131 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.752948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:17.754136 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:17.762160 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1239/2466] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:17.766280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.775808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf64ef5-2f29-458b-bb6c-8c2dabffa239 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.780302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.780919 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.781193 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.781457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.782088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:41:17.782384 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.783965 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.795734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1228/2467] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:41:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:41:17.799805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.810492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb57706-2e59-4c74-8a83-0b0bc7d0dfd9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.816033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.816671 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.816965 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.817265 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.817906 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.819815 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.846335 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1240/2468] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:17 2022] POST /identity/v3/projects => generated 401 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:17.851088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.861385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53976bf-92f8-4b0b-83f8-7472dad27d5b admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.915647 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.916237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.916509 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.917307 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.917928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.919827 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.941025 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1229/2469] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:17 2022] POST /identity/v3/users => generated 301 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:17.945092 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.955477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d304bc-39e9-4315-9278-5e07163eb1a7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.960085 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.960727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.960994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.961257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.961882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.963973 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:17.973774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1241/2470] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:17.977981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:17.987869 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99773add-cfaf-46d0-94aa-130cbb7d6666 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:17.992374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dd2664-4ce9-4d14-93d9-413256bf133d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:17.992986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd2664-4ce9-4d14-93d9-413256bf133d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:17.993318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd2664-4ce9-4d14-93d9-413256bf133d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:17.993588 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd2664-4ce9-4d14-93d9-413256bf133d admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:17.994215 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dd2664-4ce9-4d14-93d9-413256bf133d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:17.996552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dd2664-4ce9-4d14-93d9-413256bf133d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.006912 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1230/2471] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:18.011429 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.021378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c54b4a-b0ae-4d5f-8a2f-7a73254b40db admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.026025 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.026666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.026938 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.027204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] PATH_INFO: `/v3/projects/e44ca7e2d85f44dd97f8ab12aec95b9b/users/18cd19bff26e49e98f79c73fa3a44bd5/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.041296 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e44ca7e2d85f44dd97f8ab12aec95b9b, user_id=18cd19bff26e49e98f79c73fa3a44bd5, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.043994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.057834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1242/2472] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:18 2022] PUT /identity/v3/projects/e44ca7e2d85f44dd97f8ab12aec95b9b/users/18cd19bff26e49e98f79c73fa3a44bd5/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:18.068055 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.068340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.068610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.095888 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] MFA Rules not processed for user `18cd19bff26e49e98f79c73fa3a44bd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:18.097221 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:18.141364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:18.160887 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1231/2473] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:18 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:18.165503 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.226259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fadff19-69bd-4a51-b15e-3269b38cdcc7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.231648 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.232191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.232468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.232763 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.233390 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.235278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.264170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1243/2474] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:18 2022] POST /identity/v3/projects => generated 399 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:18.265596 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.277529 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31476fb6-834e-4ad3-9d26-236ac6d6915b None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.334626 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.335246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.335520 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.335790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.336420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.338379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.359687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1232/2475] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:18 2022] POST /identity/v3/users => generated 306 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:18.363708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.373864 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311b82c2-b28e-4616-acda-36c3eaa591d9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.378684 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.379280 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.379560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.379879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.380513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.382137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.393422 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1244/2476] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:18.397530 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.408126 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6223f424-0936-4d4b-8db5-76d16a035828 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.412744 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.413385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.413662 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.413930 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] PATH_INFO: `/v3/projects/d82c1717ec794a718e3d1b37c1b7fcea/users/239d8eabc7964b78a5dd1d47bcbb20b8/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.428332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d82c1717ec794a718e3d1b37c1b7fcea, user_id=239d8eabc7964b78a5dd1d47bcbb20b8, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.431037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.443211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1233/2477] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:18 2022] PUT /identity/v3/projects/d82c1717ec794a718e3d1b37c1b7fcea/users/239d8eabc7964b78a5dd1d47bcbb20b8/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:18.447246 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.457607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517e3eb-82a3-4a77-9f92-4dd17357a546 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.512843 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5146a14f-3079-4008-b25b-c5d0cee81ddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.513375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146a14f-3079-4008-b25b-c5d0cee81ddd admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.513650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146a14f-3079-4008-b25b-c5d0cee81ddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.513917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146a14f-3079-4008-b25b-c5d0cee81ddd admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.514534 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5146a14f-3079-4008-b25b-c5d0cee81ddd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.516115 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5146a14f-3079-4008-b25b-c5d0cee81ddd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.525656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1245/2478] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:18 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:18.530341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.541896 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4ca561-ee1a-4938-a201-8a8212d2e382 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.546607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.547241 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.547512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.547782 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] PATH_INFO: `/v3/projects/d82c1717ec794a718e3d1b37c1b7fcea/users/239d8eabc7964b78a5dd1d47bcbb20b8/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.549684 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d82c1717ec794a718e3d1b37c1b7fcea, user_id=239d8eabc7964b78a5dd1d47bcbb20b8, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.552293 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.562932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1234/2479] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:18 2022] PUT /identity/v3/projects/d82c1717ec794a718e3d1b37c1b7fcea/users/239d8eabc7964b78a5dd1d47bcbb20b8/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:18.574783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.575281 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.575391 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.583302 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:18.603887 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] MFA Rules not processed for user `239d8eabc7964b78a5dd1d47bcbb20b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:18.605184 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:18.671134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1246/2480] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:18 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 12 02:41:18.681634 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.683379 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:18.739712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3266a6a8-1aff-47b4-aa7e-a751ee155663 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.743258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.743840 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.744110 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.744374 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.754121 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.755532 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.765451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1235/2481] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:18 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:18.840936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:18.851145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfafbecd-a261-477c-9969-a9bbb19e3c4d None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:18.854768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3b42ef-3790-49b5-9d1f-87b2b0863e2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:18.855367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3b42ef-3790-49b5-9d1f-87b2b0863e2f service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:18.855643 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3b42ef-3790-49b5-9d1f-87b2b0863e2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:18.855912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3b42ef-3790-49b5-9d1f-87b2b0863e2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:18.914338 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3b42ef-3790-49b5-9d1f-87b2b0863e2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:18.915742 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3b42ef-3790-49b5-9d1f-87b2b0863e2f service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:18.924570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1247/2482] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:18 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:22.985765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:23.030233 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:23.048281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0ade28-16a6-40d4-8f64-8f631837bc12 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:23.052817 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:23.053433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:23.053716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:23.053981 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] PATH_INFO: `/v3/users/18cd19bff26e49e98f79c73fa3a44bd5` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:23.055090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18cd19bff26e49e98f79c73fa3a44bd5)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:23.057037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:23.106069 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18cd19bff26e49e98f79c73fa3a44bd5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.118714 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18cd19bff26e49e98f79c73fa3a44bd5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.119408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18cd19bff26e49e98f79c73fa3a44bd5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.126628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18cd19bff26e49e98f79c73fa3a44bd5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.127256 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18cd19bff26e49e98f79c73fa3a44bd5'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.140001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1236/2483] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:22 2022] DELETE /identity/v3/users/18cd19bff26e49e98f79c73fa3a44bd5 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:41:23.144416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:23.144877 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:23.144992 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:23.164605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:23.165814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:23.225701 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1248/2484] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:23.235400 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:23.236799 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:23.293849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4ec0a-677c-4949-88ba-69bc9b672df0 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:23.297438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:23.298032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:23.298303 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:23.298565 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:23.307992 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:23.309517 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:23.318838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1237/2485] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:23.581386 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:23.582807 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:23.601325 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1dda23-28e7-4027-bb9b-3a755bbfff38 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:23.606793 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:23.607435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:23.607676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:23.607944 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] PATH_INFO: `/v3/projects/e44ca7e2d85f44dd97f8ab12aec95b9b` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:23.609123 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e44ca7e2d85f44dd97f8ab12aec95b9b)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:23.611323 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:23.636366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:23.659193 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e44ca7e2d85f44dd97f8ab12aec95b9b', 'request_id': 'req-26916ca6-9e1a-45ca-9f18-274ebe6eb809'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.664607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e44ca7e2d85f44dd97f8ab12aec95b9b', 'request_id': 'req-26916ca6-9e1a-45ca-9f18-274ebe6eb809'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.672113 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e44ca7e2d85f44dd97f8ab12aec95b9b', 'request_id': 'req-26916ca6-9e1a-45ca-9f18-274ebe6eb809'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.684033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] The token cache is being invalidate because project e44ca7e2d85f44dd97f8ab12aec95b9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:23.684312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.684966 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.686461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1249/2486] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:23 2022] DELETE /identity/v3/projects/e44ca7e2d85f44dd97f8ab12aec95b9b => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:23.690414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:23.699969 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad48688b-ed3c-4c95-875c-99571eae0739 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:23.755702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:23.756247 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:23.756521 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:23.756889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] PATH_INFO: `/v3/users/239d8eabc7964b78a5dd1d47bcbb20b8` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:23.757995 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=239d8eabc7964b78a5dd1d47bcbb20b8)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:23.759916 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:23.764361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:23.805673 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '239d8eabc7964b78a5dd1d47bcbb20b8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.818059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '239d8eabc7964b78a5dd1d47bcbb20b8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.818704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '239d8eabc7964b78a5dd1d47bcbb20b8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.826117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '239d8eabc7964b78a5dd1d47bcbb20b8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.826760 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-f1ff6785-54bb-4871-9c20-b36453f2b6e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '239d8eabc7964b78a5dd1d47bcbb20b8'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:23.838886 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1238/2487] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:23 2022] DELETE /identity/v3/users/239d8eabc7964b78a5dd1d47bcbb20b8 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:24.100090 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.101840 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:24.125196 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26916ca6-9e1a-45ca-9f18-274ebe6eb809 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.181310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.181917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.182216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.182482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] PATH_INFO: `/v3/projects/d82c1717ec794a718e3d1b37c1b7fcea` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.183574 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d82c1717ec794a718e3d1b37c1b7fcea)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.185517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.230021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd82c1717ec794a718e3d1b37c1b7fcea', 'request_id': 'req-4cd53606-6cb7-4389-bc3f-ae5e96141009'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:24.235646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd82c1717ec794a718e3d1b37c1b7fcea', 'request_id': 'req-4cd53606-6cb7-4389-bc3f-ae5e96141009'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:24.243488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd82c1717ec794a718e3d1b37c1b7fcea', 'request_id': 'req-4cd53606-6cb7-4389-bc3f-ae5e96141009'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:24.254725 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] The token cache is being invalidate because project d82c1717ec794a718e3d1b37c1b7fcea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:24.255013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:24.255641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4cd53606-6cb7-4389-bc3f-ae5e96141009 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:24.257151 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1250/2488] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:24 2022] DELETE /identity/v3/projects/d82c1717ec794a718e3d1b37c1b7fcea => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:24.262170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.262648 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.262746 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.282645 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:24.283765 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:24.341536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1239/2489] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:24.349310 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.349595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.349869 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.369544 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:24.370747 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:24.378752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1251/2490] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:24.382932 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.393049 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fd2ba9-ae48-4c87-b2c5-967649b0007f None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.397636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.398222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.398500 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.398769 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.399404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:41:24.399704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.401968 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.413484 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1240/2491] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:41:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:41:24.417578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.426626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedce67d-5763-4cf5-ab9b-26b5d4b38b4c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.431300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.431886 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.432161 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.432425 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.433087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.434954 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.461451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1252/2492] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/projects => generated 379 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:41:24.466177 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.476151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6e691-7959-47b9-81de-455e3dbba033 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.531393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.531994 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.532289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.532558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.533259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.535117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.551216 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:24.559414 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1241/2493] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/users => generated 290 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:24.560611 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.570884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71378c0-9889-4cfb-a53a-17391fb96cb9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.576031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.576626 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.576933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.577203 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.577839 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.579426 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.589781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1253/2494] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:24.593536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.603562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c89a80-fd23-49bd-b638-1e040cec6e65 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.608131 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199ec6c3-8687-4694-94a3-f4364dbf038c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.608741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ec6c3-8687-4694-94a3-f4364dbf038c admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.609022 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ec6c3-8687-4694-94a3-f4364dbf038c admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.609292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ec6c3-8687-4694-94a3-f4364dbf038c admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.609931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199ec6c3-8687-4694-94a3-f4364dbf038c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.611552 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199ec6c3-8687-4694-94a3-f4364dbf038c admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.622963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1242/2495] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:24.626804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.631957 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:24.638619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660b7062-379c-4816-99ba-b8b2bd0f4a16 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.644230 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.644905 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.645180 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.645448 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] PATH_INFO: `/v3/projects/3222a36f21dd4b02ae728a94c9feae2d/users/2ecce5b58ea0416e871b34c7ea4b3460/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.660850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3222a36f21dd4b02ae728a94c9feae2d, user_id=2ecce5b58ea0416e871b34c7ea4b3460, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.663587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.676566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1254/2496] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:24 2022] PUT /identity/v3/projects/3222a36f21dd4b02ae728a94c9feae2d/users/2ecce5b58ea0416e871b34c7ea4b3460/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:41:24.687561 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.687844 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.688111 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.716390 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] MFA Rules not processed for user `2ecce5b58ea0416e871b34c7ea4b3460`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:24.717615 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:24.780234 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1243/2497] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:24.784804 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.846170 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d298f4-13af-4e8c-b54c-cd6fa7d52336 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.851048 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.851646 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.851921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.852191 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.852855 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.855370 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.882358 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1255/2498] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/projects => generated 379 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:24.886536 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.899605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f7bd74-a8db-4e06-92a6-fe290bfbb423 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.955344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.955940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.956214 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:24.956480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:24.957114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:24.959030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:24.980523 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1244/2499] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:24 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:41:24.984629 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:24.994619 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edb4182-31ce-421b-8cde-d92a36d423c6 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:24.999072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:24.999659 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:24.999930 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.000195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.000862 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.002437 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.012801 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1256/2500] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:25.016609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.026900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a1d9e5-0a5f-4f56-a4d4-7d8032613f25 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.031446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec1fbba-e403-45a6-9dd1-9e2c9b7da295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.032023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec1fbba-e403-45a6-9dd1-9e2c9b7da295 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.032297 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec1fbba-e403-45a6-9dd1-9e2c9b7da295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.032564 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec1fbba-e403-45a6-9dd1-9e2c9b7da295 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.033222 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec1fbba-e403-45a6-9dd1-9e2c9b7da295 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.034885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec1fbba-e403-45a6-9dd1-9e2c9b7da295 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.045090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1245/2501] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:25.048865 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.058603 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e1c706-1f44-4486-a4e1-141091ae03c5 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.063143 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.063769 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.064061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.064327 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] PATH_INFO: `/v3/projects/e18dc8081d9b40559bcbb114d9b8454c/users/2244d714913d4b2790875972b51aab06/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.079087 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e18dc8081d9b40559bcbb114d9b8454c, user_id=2244d714913d4b2790875972b51aab06, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.081829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.095068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1257/2502] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:25 2022] PUT /identity/v3/projects/e18dc8081d9b40559bcbb114d9b8454c/users/2244d714913d4b2790875972b51aab06/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:25.157918 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.158206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.158473 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.185646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] MFA Rules not processed for user `2244d714913d4b2790875972b51aab06`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:25.186849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:25.231496 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:25.251436 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1246/2503] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:25.256036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.317479 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b890f3fb-8afd-4ea9-a905-f392ddf056a7 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.322061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.322660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.323473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.323739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.324369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.326268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.352278 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1258/2504] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:25 2022] POST /identity/v3/projects => generated 377 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:25.356290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.366786 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c9f15e-4c64-4b1c-911e-245c6ac67f7a None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.422205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.422823 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.423095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.423357 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.423976 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.425920 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.446607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1247/2505] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:25 2022] POST /identity/v3/users => generated 295 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:25.451223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.461270 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b5e8f-f83c-4dc2-bdce-7707277b9d15 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.466468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.467055 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.467330 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.467598 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.468223 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.469851 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.480493 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1259/2506] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:25.484406 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.494538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bed8d5-0a25-4039-9dd7-23263da7f538 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.499636 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.500269 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.500543 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.500855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] PATH_INFO: `/v3/projects/661c108e8fca4891b461378205189475/users/088d0fe088ec4469aa181fd86727b707/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.516290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=661c108e8fca4891b461378205189475, user_id=088d0fe088ec4469aa181fd86727b707, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.519280 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.531528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1248/2507] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:25 2022] PUT /identity/v3/projects/661c108e8fca4891b461378205189475/users/088d0fe088ec4469aa181fd86727b707/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:25.535349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.540467 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:25.546496 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b8ad04-2592-4292-99b2-8c734025d281 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.601812 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b0b2f-f879-4a07-a489-7c924864f818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.602365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b0b2f-f879-4a07-a489-7c924864f818 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.602637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b0b2f-f879-4a07-a489-7c924864f818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.602901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b0b2f-f879-4a07-a489-7c924864f818 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.603518 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6b0b2f-f879-4a07-a489-7c924864f818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.605145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6b0b2f-f879-4a07-a489-7c924864f818 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.614469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1260/2508] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:41:25.619056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.629681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755c3414-8425-464a-859f-4cbcf7e85763 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.634338 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.635033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.635908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.636207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] PATH_INFO: `/v3/projects/661c108e8fca4891b461378205189475/users/088d0fe088ec4469aa181fd86727b707/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.638166 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=661c108e8fca4891b461378205189475, user_id=088d0fe088ec4469aa181fd86727b707, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.640812 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.650952 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1249/2509] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:25 2022] PUT /identity/v3/projects/661c108e8fca4891b461378205189475/users/088d0fe088ec4469aa181fd86727b707/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:25.661046 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.661332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.661600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.689321 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] MFA Rules not processed for user `088d0fe088ec4469aa181fd86727b707`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:25.690543 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:25.758154 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1261/2510] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:25 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:25.770284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.772041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:25.828425 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b77c1f-7efa-4718-834d-d30030a82cbf admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.832072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.832659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.832971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.833237 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.842433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.844170 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.855015 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1250/2511] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:25 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:25.896183 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:25.897597 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:25.910633 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f8c12-8238-4944-a1bf-9bf7c8ffd00c None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:25.914582 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836488b8-a0fa-47ed-a875-26d69df8a1af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:25.915175 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836488b8-a0fa-47ed-a875-26d69df8a1af service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:25.915450 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836488b8-a0fa-47ed-a875-26d69df8a1af service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:25.915723 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836488b8-a0fa-47ed-a875-26d69df8a1af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:25.971403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836488b8-a0fa-47ed-a875-26d69df8a1af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:25.972857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836488b8-a0fa-47ed-a875-26d69df8a1af service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:25.982682 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1262/2512] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:25 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:28.846913 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:28.891306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:28.909907 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b0f8a9-5b12-437f-95bd-595709f077f2 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:28.914685 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:28.915299 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:28.915922 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:28.916191 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] PATH_INFO: `/v3/users/2ecce5b58ea0416e871b34c7ea4b3460` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:28.918144 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ecce5b58ea0416e871b34c7ea4b3460)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:28.920287 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:28.965382 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ecce5b58ea0416e871b34c7ea4b3460'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:28.977593 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ecce5b58ea0416e871b34c7ea4b3460'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:28.978258 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ecce5b58ea0416e871b34c7ea4b3460'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:28.985971 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ecce5b58ea0416e871b34c7ea4b3460'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:28.986605 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ecce5b58ea0416e871b34c7ea4b3460'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:28.999362 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1251/2513] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:28 2022] DELETE /identity/v3/users/2ecce5b58ea0416e871b34c7ea4b3460 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:29.003836 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:29.004342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:29.004342 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:29.023825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:29.025079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:29.041197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=100 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:29.082892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1263/2514] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:29.092928 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:29.094306 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:29.156728 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886fc08c-42a9-48cb-8874-70a0eca7983a admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:29.162072 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:29.162683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:29.162955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:29.163219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:29.172975 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:29.174393 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:29.185211 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1252/2515] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:29.452764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:29.454257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:29.470378 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0122042-c9ef-4e46-8a62-a6386d739dc5 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:29.475489 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:29.476400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:29.476676 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:29.477100 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] PATH_INFO: `/v3/projects/3222a36f21dd4b02ae728a94c9feae2d` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:29.478110 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] RBAC: Authorizing `identity:delete_project(project_id=3222a36f21dd4b02ae728a94c9feae2d)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:29.480849 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:29.526192 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3222a36f21dd4b02ae728a94c9feae2d', 'request_id': 'req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.532117 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3222a36f21dd4b02ae728a94c9feae2d', 'request_id': 'req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.538845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3222a36f21dd4b02ae728a94c9feae2d', 'request_id': 'req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.550749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] The token cache is being invalidate because project 3222a36f21dd4b02ae728a94c9feae2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:29.551029 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.551638 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.553129 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1264/2516] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:29 2022] DELETE /identity/v3/projects/3222a36f21dd4b02ae728a94c9feae2d => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:29.557139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:29.566512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bec606-2222-4d1b-bec4-8c123fe609fe service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:29.622474 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:29.623087 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:29.623360 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:29.623623 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] PATH_INFO: `/v3/users/2244d714913d4b2790875972b51aab06` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:29.624726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=2244d714913d4b2790875972b51aab06)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:29.626651 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:29.631597 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:29.673735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2244d714913d4b2790875972b51aab06'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.686361 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2244d714913d4b2790875972b51aab06'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.687040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2244d714913d4b2790875972b51aab06'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.694104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2244d714913d4b2790875972b51aab06'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.694721 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2244d714913d4b2790875972b51aab06'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:29.706877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1253/2517] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:29 2022] DELETE /identity/v3/users/2244d714913d4b2790875972b51aab06 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:29.973845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:29.975282 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:29.988885 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81a75b4-ad34-44e2-a0e2-7bb1e5792adf admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:30.045559 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:30.046182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.046457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.046733 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] PATH_INFO: `/v3/projects/e18dc8081d9b40559bcbb114d9b8454c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.047845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e18dc8081d9b40559bcbb114d9b8454c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:30.049792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:30.094035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:30.095516 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e18dc8081d9b40559bcbb114d9b8454c', 'request_id': 'req-280edc2b-8a7f-4826-84b1-f56387e7bd0f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.103381 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e18dc8081d9b40559bcbb114d9b8454c', 'request_id': 'req-280edc2b-8a7f-4826-84b1-f56387e7bd0f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.111570 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e18dc8081d9b40559bcbb114d9b8454c', 'request_id': 'req-280edc2b-8a7f-4826-84b1-f56387e7bd0f'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.123167 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] The token cache is being invalidate because project e18dc8081d9b40559bcbb114d9b8454c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:30.123457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.124121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.125637 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1265/2518] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:29 2022] DELETE /identity/v3/projects/e18dc8081d9b40559bcbb114d9b8454c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:30.129609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:30.131042 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:30.143590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efebc736-30a9-4df8-9172-a736021545aa admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:30.199181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:30.199795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.200066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.200328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] PATH_INFO: `/v3/users/088d0fe088ec4469aa181fd86727b707` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.201446 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] RBAC: Authorizing `identity:delete_user(user_id=088d0fe088ec4469aa181fd86727b707)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:30.203343 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:30.247815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '088d0fe088ec4469aa181fd86727b707'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.259558 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '088d0fe088ec4469aa181fd86727b707'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.260223 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '088d0fe088ec4469aa181fd86727b707'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.267285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '088d0fe088ec4469aa181fd86727b707'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.267900 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-b17c3daa-c31b-4c0d-bea2-7121743fa700 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '088d0fe088ec4469aa181fd86727b707'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.280432 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1254/2519] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:30 2022] DELETE /identity/v3/users/088d0fe088ec4469aa181fd86727b707 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:30.552485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:30.554020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:30.569666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280edc2b-8a7f-4826-84b1-f56387e7bd0f admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:30.626847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:30.627474 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.627738 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.628022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] PATH_INFO: `/v3/projects/661c108e8fca4891b461378205189475` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.629187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] RBAC: Authorizing `identity:delete_project(project_id=661c108e8fca4891b461378205189475)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:30.631010 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:30.677408 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '661c108e8fca4891b461378205189475', 'request_id': 'req-a8592dac-ef66-412e-bd90-6d1eecdf2615'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.683472 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '661c108e8fca4891b461378205189475', 'request_id': 'req-a8592dac-ef66-412e-bd90-6d1eecdf2615'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.691442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '661c108e8fca4891b461378205189475', 'request_id': 'req-a8592dac-ef66-412e-bd90-6d1eecdf2615'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.702985 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] The token cache is being invalidate because project 661c108e8fca4891b461378205189475 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81364) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:30.703269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.703921 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-a8592dac-ef66-412e-bd90-6d1eecdf2615 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:30.705461 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1266/2520] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:30 2022] DELETE /identity/v3/projects/661c108e8fca4891b461378205189475 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:30.711385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.711677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.711949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.719458 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:30.733335 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:30.734551 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:30.792813 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1255/2521] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:30.802163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.802447 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.802810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.822731 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:30.824464 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:30.832318 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1267/2522] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:30.836493 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:30.846674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee84428-ab71-44c2-bf22-e078e6cbe983 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:30.851450 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:30.852032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.852301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.852562 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] PATH_INFO: `/v3/domains` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.853219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81365) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:41:30.853512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:30.855093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:30.866512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1256/2523] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:41:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:41:30.871380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:30.880628 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b389a92-bbc2-4e70-aaa1-aea3191a9d52 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:30.885239 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:30.886352 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.886654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.886904 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.887529 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:30.889443 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:30.916360 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1268/2524] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:30 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:30.920355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:30.930550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e7a0d3-cb55-4c90-81a4-888943d6aa77 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:30.986266 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:30.986863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:30.987139 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:30.987409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:30.988030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:30.990054 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:31.011573 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1257/2525] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:30 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:41:31.015696 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:31.025958 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edbbf2-a3d5-4830-9609-a59b470635d9 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:31.031108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:31.032112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:31.032578 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:31.033177 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:31.034001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:31.036810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:31.047857 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1269/2526] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 12 02:41:31.051867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:31.062219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db62db5b-4bf6-426f-b53d-7e010f5e6d20 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:31.066701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58736b4c-6107-4683-9c7f-c486beea5201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:31.067285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58736b4c-6107-4683-9c7f-c486beea5201 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:31.067578 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58736b4c-6107-4683-9c7f-c486beea5201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:31.067818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58736b4c-6107-4683-9c7f-c486beea5201 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:31.068444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58736b4c-6107-4683-9c7f-c486beea5201 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:31.070099 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58736b4c-6107-4683-9c7f-c486beea5201 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:31.081726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1258/2527] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:31.085698 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:31.095495 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605177b-aced-4556-a16f-57a83ad99a7e admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:31.100345 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:31.101317 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:31.101591 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:31.101859 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] PATH_INFO: `/v3/projects/6be843ac99544b94a880fd03435698a1/users/3451249b1d7c4134a31a04220bfc8a21/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:31.117049 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=6be843ac99544b94a880fd03435698a1, user_id=3451249b1d7c4134a31a04220bfc8a21, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:31.119694 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:31.132979 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1270/2528] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:31 2022] PUT /identity/v3/projects/6be843ac99544b94a880fd03435698a1/users/3451249b1d7c4134a31a04220bfc8a21/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:31.137972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ff97f1-ccc7-463e-875d-4542c4b5ab78 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:31.138422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ff97f1-ccc7-463e-875d-4542c4b5ab78 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:31.138538 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ff97f1-ccc7-463e-875d-4542c4b5ab78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:31.156876 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-27ff97f1-ccc7-463e-875d-4542c4b5ab78 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:31.158057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-27ff97f1-ccc7-463e-875d-4542c4b5ab78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:31.216511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1259/2529] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:31.226455 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:31.228210 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:31.234038 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:31.286933 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e27fea-eadf-4ce3-941c-64750baf72fd admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:31.290654 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:31.291257 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:31.291531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:31.291795 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:31.301153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:31.302567 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:31.312332 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1271/2530] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:41:31.925682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:31.926011 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:31.926279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:31.946709 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:31.959712 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] MFA Rules not processed for user `3451249b1d7c4134a31a04220bfc8a21`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:31.961024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:32.028452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1260/2531] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:32.033005 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:32.042560 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baedf5ae-4452-49b0-9af4-fdd89d04aac8 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:32.047093 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:32.047679 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.047952 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.048494 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:32.048872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:32.050764 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:32.079047 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1272/2532] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:32 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:32.083062 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:32.094314 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7045cc-fdc7-4809-9b26-1d98b853cf23 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:32.155024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:32.155628 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.155903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.156171 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] PATH_INFO: `/v3/users` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:32.156832 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:32.158741 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:32.179727 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1261/2533] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:32 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:41:32.184222 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:32.194917 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612bda61-5aaf-479f-b6f3-ed05b9736fe8 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:32.200128 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:32.200662 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.200962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.201226 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:32.201845 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:32.203416 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:32.214219 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1273/2534] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:32.218195 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:32.228246 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e05ecf9-debb-4a09-96cd-03cf9009276d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:32.232792 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:32.233419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.233687 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.233949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] PATH_INFO: `/v3/projects/0d26e187d4db4d288a2d651093be5a03/users/9c8dc82f5f6e4728b5b8b9c61c94113c/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:32.248768 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d26e187d4db4d288a2d651093be5a03, user_id=9c8dc82f5f6e4728b5b8b9c61c94113c, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:32.251438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:32.264057 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1262/2535] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:32 2022] PUT /identity/v3/projects/0d26e187d4db4d288a2d651093be5a03/users/9c8dc82f5f6e4728b5b8b9c61c94113c/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:32.267928 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:32.277818 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc769e7-2a0a-4587-b0f2-caddf86dd060 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:32.333778 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3647c5-f198-495a-b093-f913e6f15bad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:32.334380 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3647c5-f198-495a-b093-f913e6f15bad admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.334650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3647c5-f198-495a-b093-f913e6f15bad admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.334913 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3647c5-f198-495a-b093-f913e6f15bad admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:32.335532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3647c5-f198-495a-b093-f913e6f15bad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:32.337137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3647c5-f198-495a-b093-f913e6f15bad admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:32.346691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1274/2536] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:32.351164 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:32.361281 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210d93d4-c8b2-421a-a84b-c8e8bc6abbbf admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:32.365880 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:32.366510 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.366784 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.367051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] PATH_INFO: `/v3/projects/0d26e187d4db4d288a2d651093be5a03/users/9c8dc82f5f6e4728b5b8b9c61c94113c/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:32.368960 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d26e187d4db4d288a2d651093be5a03, user_id=9c8dc82f5f6e4728b5b8b9c61c94113c, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:32.372066 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:32.382086 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1263/2537] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:32 2022] PUT /identity/v3/projects/0d26e187d4db4d288a2d651093be5a03/users/9c8dc82f5f6e4728b5b8b9c61c94113c/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:32.986492 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:32.986776 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:32.987042 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:33.015533 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] MFA Rules not processed for user `9c8dc82f5f6e4728b5b8b9c61c94113c`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:33.016832 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:33.077556 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:33.090842 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1275/2538] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:32 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:33.102682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:33.104363 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:33.163708 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d71d1-364d-4830-82f5-d8633b4bd12f admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:33.167849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:33.168455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:33.168752 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:33.169020 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:33.173186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:33.226352 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:33.227788 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:33.237344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1264/2539] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:34.591109 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:34.592501 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:34.605341 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba06173-8230-4265-95e0-2cfca0daa08e None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:34.608894 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:34.609486 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:34.609762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:34.610033 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:34.618816 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:34.620232 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:34.629482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1276/2540] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:34 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:43.233790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=25 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:44.625372 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=9 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:46.988094 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:46.989716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:47.063806 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26f9f33-f74e-4156-ad4c-d67eec5c42f7 service cinder] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:47.069119 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:47.069793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:47.070071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:47.070340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:47.072082 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:47.084271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:47.085903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:47.100939 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1265/2541] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:47.775247 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:47.839660 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134a388c-e71f-4944-8787-90fb92e4d0ed service cinder] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:47.845078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:47.845802 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:47.846112 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:47.846453 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] PATH_INFO: `/v3/users/3451249b1d7c4134a31a04220bfc8a21` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:47.847635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3451249b1d7c4134a31a04220bfc8a21)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:47.849708 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:47.894410 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3451249b1d7c4134a31a04220bfc8a21'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:47.906752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3451249b1d7c4134a31a04220bfc8a21'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:47.907525 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3451249b1d7c4134a31a04220bfc8a21'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:47.914783 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3451249b1d7c4134a31a04220bfc8a21'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:47.915406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3451249b1d7c4134a31a04220bfc8a21'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:47.928101 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1277/2542] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:47 2022] DELETE /identity/v3/users/3451249b1d7c4134a31a04220bfc8a21 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:48.102225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:48.114048 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6125909c-89bd-4bfc-a64a-b236b44fcf33 service placement] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:48.172680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:48.173271 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.173546 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.173816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] PATH_INFO: `/v3/projects/6be843ac99544b94a880fd03435698a1` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.174914 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6be843ac99544b94a880fd03435698a1)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:48.176915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:48.221449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6be843ac99544b94a880fd03435698a1', 'request_id': 'req-ae288bb0-bf1e-4c09-8175-5f13df83d381'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.227093 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6be843ac99544b94a880fd03435698a1', 'request_id': 'req-ae288bb0-bf1e-4c09-8175-5f13df83d381'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.233227 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6be843ac99544b94a880fd03435698a1', 'request_id': 'req-ae288bb0-bf1e-4c09-8175-5f13df83d381'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.244739 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] The token cache is being invalidate because project 6be843ac99544b94a880fd03435698a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:48.245023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.245646 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.247097 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1266/2543] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:48 2022] DELETE /identity/v3/projects/6be843ac99544b94a880fd03435698a1 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:48.251256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:48.252745 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:48.266484 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af586f-5e04-4229-b00e-4be012403f08 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:48.323872 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:48.324502 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.324831 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.325105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] PATH_INFO: `/v3/users/9c8dc82f5f6e4728b5b8b9c61c94113c` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.326241 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c8dc82f5f6e4728b5b8b9c61c94113c)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:48.328647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:48.351635 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:48.375088 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c8dc82f5f6e4728b5b8b9c61c94113c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.386527 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c8dc82f5f6e4728b5b8b9c61c94113c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.387182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c8dc82f5f6e4728b5b8b9c61c94113c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.394959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c8dc82f5f6e4728b5b8b9c61c94113c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.395752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-69d4f435-7e0b-4c3e-81ae-1346f42745dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c8dc82f5f6e4728b5b8b9c61c94113c'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.408587 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1278/2544] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:48 2022] DELETE /identity/v3/users/9c8dc82f5f6e4728b5b8b9c61c94113c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:48.574059 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:48.575528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:48.588790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae288bb0-bf1e-4c09-8175-5f13df83d381 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:48.606550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:48.648775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:48.649402 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.649677 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.649948 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] PATH_INFO: `/v3/projects/0d26e187d4db4d288a2d651093be5a03` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.651053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d26e187d4db4d288a2d651093be5a03)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:48.653071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:48.698207 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d26e187d4db4d288a2d651093be5a03', 'request_id': 'req-1c13ac84-ccdb-4eae-bd0d-543e289064ca'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.704404 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d26e187d4db4d288a2d651093be5a03', 'request_id': 'req-1c13ac84-ccdb-4eae-bd0d-543e289064ca'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.710620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d26e187d4db4d288a2d651093be5a03', 'request_id': 'req-1c13ac84-ccdb-4eae-bd0d-543e289064ca'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.722681 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] The token cache is being invalidate because project 0d26e187d4db4d288a2d651093be5a03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:48.722989 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.723598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-1c13ac84-ccdb-4eae-bd0d-543e289064ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:48.725106 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1267/2545] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:48 2022] DELETE /identity/v3/projects/0d26e187d4db4d288a2d651093be5a03 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:48.730498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.730781 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.731053 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.751188 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:48.752387 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:48.808720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1279/2546] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:48.817586 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.817904 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.818176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.838240 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:48.839420 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:48.848310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1268/2547] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:48.852911 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:48.862700 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af41d24e-6544-48ee-8e8a-3812ed099d12 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:48.867267 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:48.867858 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.868134 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.868400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.869069 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:41:48.869366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:48.870948 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:48.882936 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1280/2548] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:41:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:41:48.887090 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:48.897071 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0dfc7-72b5-4463-93ff-f6ea7f9480c6 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:48.901740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:48.902328 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:48.902607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:48.902883 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:48.903507 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:48.905444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:48.934337 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1269/2549] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:48 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:48.938491 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:48.948336 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5cecea-b230-4354-8916-7166bcaa1d56 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:49.003483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:49.004020 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:49.004297 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:49.004568 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:49.005303 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:49.007212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:49.028539 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1281/2550] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:48 2022] POST /identity/v3/users => generated 292 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 12 02:41:49.033775 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:49.044301 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3276102-959e-4439-a0cd-b9f8af7eb675 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:49.049026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:49.049607 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:49.049885 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:49.050154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:49.050818 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:49.052411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:49.062369 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1270/2551] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:49.066242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:49.076268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6bb33e-82e2-4ee9-a660-856b15cb3755 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:49.081348 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76c268e-905d-4719-b7bc-b909e5cba46d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:49.081924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76c268e-905d-4719-b7bc-b909e5cba46d admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:49.082195 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76c268e-905d-4719-b7bc-b909e5cba46d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:49.082460 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76c268e-905d-4719-b7bc-b909e5cba46d admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:49.083078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76c268e-905d-4719-b7bc-b909e5cba46d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:49.084645 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76c268e-905d-4719-b7bc-b909e5cba46d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:49.095834 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1282/2552] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:49.099838 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:49.110178 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6464e-b06c-48d8-90b3-94e35016e2be admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:49.115155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:49.115879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:49.116219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:49.116492 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] PATH_INFO: `/v3/projects/ca82c0afaef241b1bff14d2f2cf13de9/users/af76b24a13024ec1a90f0a2892f275c1/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:49.132556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca82c0afaef241b1bff14d2f2cf13de9, user_id=af76b24a13024ec1a90f0a2892f275c1, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:49.135572 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:49.149470 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1271/2553] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:49 2022] PUT /identity/v3/projects/ca82c0afaef241b1bff14d2f2cf13de9/users/af76b24a13024ec1a90f0a2892f275c1/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:49.160298 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:49.160579 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:49.160882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:49.168847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:49.188959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] MFA Rules not processed for user `af76b24a13024ec1a90f0a2892f275c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:49.190197 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:49.253319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1283/2554] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:49.264245 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:49.266537 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:49.319471 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:49.325480 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19224f2e-f2c3-4acf-bc1b-b8cbdc3be637 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:49.330815 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924d2d25-0862-436a-8f75-ee16cac33f72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:49.331412 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924d2d25-0862-436a-8f75-ee16cac33f72 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:49.331683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924d2d25-0862-436a-8f75-ee16cac33f72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:49.331949 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924d2d25-0862-436a-8f75-ee16cac33f72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:49.340449 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924d2d25-0862-436a-8f75-ee16cac33f72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:49.341955 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924d2d25-0862-436a-8f75-ee16cac33f72 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:49.351740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1272/2555] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:56.693269 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:56.754477 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87072d2b-4394-48a4-9472-c83751ceabbb None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:56.758980 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:56.759608 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:56.759882 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:56.760153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] PATH_INFO: `/v3/users/af76b24a13024ec1a90f0a2892f275c1` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:56.761324 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af76b24a13024ec1a90f0a2892f275c1)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:56.763243 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:56.807710 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af76b24a13024ec1a90f0a2892f275c1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:56.820288 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af76b24a13024ec1a90f0a2892f275c1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:56.820975 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af76b24a13024ec1a90f0a2892f275c1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:56.827935 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af76b24a13024ec1a90f0a2892f275c1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:56.828554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af76b24a13024ec1a90f0a2892f275c1'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:56.840900 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1284/2556] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:41:56 2022] DELETE /identity/v3/users/af76b24a13024ec1a90f0a2892f275c1 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:56.846225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:56.846770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:56.846770 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:56.866341 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:56.867600 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:56.924967 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1273/2557] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:56.934511 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:56.935898 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:56.950077 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:56.996201 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969e4fba-5cc0-46dd-9232-934333834a77 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:56.999968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79803264-7582-4dcf-973e-881ffff5f6ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.000572 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79803264-7582-4dcf-973e-881ffff5f6ab service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.000878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79803264-7582-4dcf-973e-881ffff5f6ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.001171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79803264-7582-4dcf-973e-881ffff5f6ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.010103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79803264-7582-4dcf-973e-881ffff5f6ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.011507 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79803264-7582-4dcf-973e-881ffff5f6ab service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.024490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1285/2558] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:41:57.290868 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.292383 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:57.305037 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61a606-c6a8-461d-bbd1-be3e42346758 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.309734 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.310457 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.310736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.311007 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] PATH_INFO: `/v3/projects/ca82c0afaef241b1bff14d2f2cf13de9` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.312186 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca82c0afaef241b1bff14d2f2cf13de9)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.314320 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.360424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca82c0afaef241b1bff14d2f2cf13de9', 'request_id': 'req-43803dc7-1d6d-449b-937b-58df0c8aef5b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:57.365856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca82c0afaef241b1bff14d2f2cf13de9', 'request_id': 'req-43803dc7-1d6d-449b-937b-58df0c8aef5b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:57.372013 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca82c0afaef241b1bff14d2f2cf13de9', 'request_id': 'req-43803dc7-1d6d-449b-937b-58df0c8aef5b'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:57.384077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] The token cache is being invalidate because project ca82c0afaef241b1bff14d2f2cf13de9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:41:57.384365 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:57.385026 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-43803dc7-1d6d-449b-937b-58df0c8aef5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:41:57.386512 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1274/2559] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:41:57 2022] DELETE /identity/v3/projects/ca82c0afaef241b1bff14d2f2cf13de9 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:57.392007 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.392286 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.392554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.412331 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:57.413554 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:57.470316 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1286/2560] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:57.479620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.479903 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.480172 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.500641 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:57.501863 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:57.509889 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1275/2561] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:57.515037 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.525130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8697bb15-dbd1-4bf0-ab20-752e08e2f0fb None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.529924 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.530531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.530835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.531130 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.531768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:41:57.532067 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.533718 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.545319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1287/2562] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:41:57.550219 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.559452 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28107996-fda0-497a-adb2-6fbbe692b3d7 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.564811 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.566001 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.566286 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.566560 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.567220 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.569175 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.595895 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1276/2563] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:41:57 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:57.599959 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.610521 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02326fdb-ade4-411b-a0f4-3fa523c6ae99 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.665469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.666071 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.666339 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.666599 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.667204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.669120 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.690340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1288/2564] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:41:57 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:41:57.694522 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.704963 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b0d4a5-c41f-4fbf-89d3-c671a8730ae1 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.709619 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.710205 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.710478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.710745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.711399 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.713116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.723569 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1277/2565] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:57.727758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.738256 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e63bb6-bbfd-4ad0-839c-25d06e616917 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.742785 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b319e9c0-00f2-4b02-b779-2bf032a7ab1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.743369 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b319e9c0-00f2-4b02-b779-2bf032a7ab1a admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.743639 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b319e9c0-00f2-4b02-b779-2bf032a7ab1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.743927 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b319e9c0-00f2-4b02-b779-2bf032a7ab1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.744562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b319e9c0-00f2-4b02-b779-2bf032a7ab1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.746217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b319e9c0-00f2-4b02-b779-2bf032a7ab1a admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.756485 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1289/2566] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:41:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:41:57.760489 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.770151 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8d2fa-7dc9-4467-8734-7372ceb6cfe4 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.774683 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.775318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.775589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.775855 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] PATH_INFO: `/v3/projects/f63b65c8231b42e4a57f35e4a1d5fec0/users/efa6a5d2f532409b96d42ba12b6204db/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.790463 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f63b65c8231b42e4a57f35e4a1d5fec0, user_id=efa6a5d2f532409b96d42ba12b6204db, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.793155 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:57.806284 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1278/2567] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:41:57 2022] PUT /identity/v3/projects/f63b65c8231b42e4a57f35e4a1d5fec0/users/efa6a5d2f532409b96d42ba12b6204db/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:41:57.817438 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.817716 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.817994 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.846001 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] MFA Rules not processed for user `efa6a5d2f532409b96d42ba12b6204db`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:41:57.847490 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:57.856878 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:57.912159 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1290/2568] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:41:57 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:41:57.923437 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:41:57.925161 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:41:57.935793 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:41:57.982682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59ec8dc-56b9-49e1-9d78-b066a745e758 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:41:57.986313 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5878cb-d3a8-4b6f-9ed1-06b48fa20d91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '575abe426dae4b868bb1c35537a9be35', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:41:57.986906 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5878cb-d3a8-4b6f-9ed1-06b48fa20d91 service cinder] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:41:57.987183 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5878cb-d3a8-4b6f-9ed1-06b48fa20d91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:41:57.987455 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5878cb-d3a8-4b6f-9ed1-06b48fa20d91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:41:57.995654 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5878cb-d3a8-4b6f-9ed1-06b48fa20d91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:41:57.997114 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5878cb-d3a8-4b6f-9ed1-06b48fa20d91 service cinder] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:41:58.005879 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1279/2569] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:41:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:42:07.909255 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=22 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:07.982513 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.001821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=22 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:08.043268 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb34922-ea7b-4bc2-9031-ecf0524e07e0 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.048366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.049031 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.049312 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.049583 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] PATH_INFO: `/v3/users/efa6a5d2f532409b96d42ba12b6204db` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.050691 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] RBAC: Authorizing `identity:delete_user(user_id=efa6a5d2f532409b96d42ba12b6204db)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:08.052607 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:08.096573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efa6a5d2f532409b96d42ba12b6204db'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.109259 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efa6a5d2f532409b96d42ba12b6204db'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.109951 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efa6a5d2f532409b96d42ba12b6204db'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.117367 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efa6a5d2f532409b96d42ba12b6204db'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.118418 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efa6a5d2f532409b96d42ba12b6204db'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.130365 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1291/2570] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:42:07 2022] DELETE /identity/v3/users/efa6a5d2f532409b96d42ba12b6204db => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:42:08.134795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.135077 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.135348 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.154038 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:08.155232 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:08.212974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1280/2571] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:08.223396 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.224829 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:08.281535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e77f30-8ac5-4b33-9aab-d95b9c72893c admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.285204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dad14df-3af9-4bb6-b18e-a19c09bcc07d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.285805 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dad14df-3af9-4bb6-b18e-a19c09bcc07d service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.286079 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dad14df-3af9-4bb6-b18e-a19c09bcc07d service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.286347 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dad14df-3af9-4bb6-b18e-a19c09bcc07d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.295163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dad14df-3af9-4bb6-b18e-a19c09bcc07d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:08.297171 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dad14df-3af9-4bb6-b18e-a19c09bcc07d service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:08.306592 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1292/2572] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:42:08.573419 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.574835 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:08.586982 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a999b42-afa2-4b51-b63e-3e9aabef278d None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.592842 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.594409 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.594772 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.595115 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] PATH_INFO: `/v3/projects/f63b65c8231b42e4a57f35e4a1d5fec0` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.596431 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f63b65c8231b42e4a57f35e4a1d5fec0)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:08.599315 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:08.648265 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f63b65c8231b42e4a57f35e4a1d5fec0', 'request_id': 'req-dd0de076-7372-4784-83c5-4a0c2774f512'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.653881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f63b65c8231b42e4a57f35e4a1d5fec0', 'request_id': 'req-dd0de076-7372-4784-83c5-4a0c2774f512'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.660667 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f63b65c8231b42e4a57f35e4a1d5fec0', 'request_id': 'req-dd0de076-7372-4784-83c5-4a0c2774f512'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.672330 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] The token cache is being invalidate because project f63b65c8231b42e4a57f35e4a1d5fec0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:42:08.672613 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.673260 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-dd0de076-7372-4784-83c5-4a0c2774f512 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:42:08.674730 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1281/2573] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:42:08 2022] DELETE /identity/v3/projects/f63b65c8231b42e4a57f35e4a1d5fec0 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:42:08.679819 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.680105 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.680375 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.699458 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:08.700650 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:08.717575 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:08.760084 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1293/2574] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:08.769599 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.769890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.770169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.790322 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:08.791533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:08.799344 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1282/2575] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:08.803432 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.813080 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda9fdf3-3a92-4d81-b5cd-33f8ba8f6460 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.817594 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.818187 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.818463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.818734 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] PATH_INFO: `/v3/domains` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.819366 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81364) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 12 02:42:08.819666 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:08.821444 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:08.833260 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1294/2576] 10.222.0.23 () {58 vars in 1153 bytes} [Wed Oct 12 02:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 12 02:42:08.837302 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.847073 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889b20ad-3a03-482e-8483-c3c9b366adb1 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.852495 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.853168 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.853444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.853708 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.854332 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:08.856236 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:08.883821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1283/2577] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:42:08 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:42:08.887908 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.898249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5970642-6849-4c26-bb8c-275467065e6d admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.953133 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.953828 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.954204 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.954473 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.955099 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:08.957021 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:08.977463 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1295/2578] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:42:08 2022] POST /identity/v3/users => generated 289 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:42:08.981700 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:08.991908 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d18d57-8fda-466a-a94e-3af109a97606 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:08.996627 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:08.997253 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:08.997528 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:08.997795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:08.998427 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:09.000550 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:09.010620 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1284/2579] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:42:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:42:09.014249 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:09.024774 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2499dd-998e-448b-89a3-1c046fb03ebc admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:09.029371 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40987d4-89b4-4e97-be61-77c85a0dae13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:09.029962 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40987d4-89b4-4e97-be61-77c85a0dae13 admin admin] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:09.030237 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40987d4-89b4-4e97-be61-77c85a0dae13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:09.030514 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40987d4-89b4-4e97-be61-77c85a0dae13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:09.031145 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40987d4-89b4-4e97-be61-77c85a0dae13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:09.032739 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40987d4-89b4-4e97-be61-77c85a0dae13 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:09.043181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1296/2580] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:42:09 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:42:09.047030 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:09.056780 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ee7385-aefe-4727-896d-53a601422e61 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:09.061292 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:09.062428 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] REQUEST_METHOD: `PUT` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:09.062699 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:09.062966 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] PATH_INFO: `/v3/projects/0420b1fe8ff5451dba85de3001a3932f/users/e184afaa0abd4e8eb1bc641167dc0b9b/roles/7914a50fd85645bd870c4f7502de5b02` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:09.077540 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0420b1fe8ff5451dba85de3001a3932f, user_id=e184afaa0abd4e8eb1bc641167dc0b9b, role_id=7914a50fd85645bd870c4f7502de5b02)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:09.081701 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:09.095460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1285/2581] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:42:09 2022] PUT /identity/v3/projects/0420b1fe8ff5451dba85de3001a3932f/users/e184afaa0abd4e8eb1bc641167dc0b9b/roles/7914a50fd85645bd870c4f7502de5b02 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:42:09.100498 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71168b15-c7e8-45f1-b1d3-fc7ffa62c982 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:09.100847 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71168b15-c7e8-45f1-b1d3-fc7ffa62c982 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:09.101104 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71168b15-c7e8-45f1-b1d3-fc7ffa62c982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:09.123108 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-71168b15-c7e8-45f1-b1d3-fc7ffa62c982 None None] MFA Rules not processed for user `1bca8e4c760d498693a8dcfc0e178249`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:09.124319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-71168b15-c7e8-45f1-b1d3-fc7ffa62c982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:09.185810 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1297/2582] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:09.195539 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:09.197254 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:09.220181 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:09.256033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eecf8d8-358f-4091-b9c0-7ec7a85bb278 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:09.259658 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:09.260259 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:09.260533 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:09.260884 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:09.270444 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:09.271867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:09.281771 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1286/2583] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:42:12.918103 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:12.918656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:12.918656 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:12.952068 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] MFA Rules not processed for user `e184afaa0abd4e8eb1bc641167dc0b9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:12.954137 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:12.972349 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:13.030968 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1298/2584] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:12 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:13.035366 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:13.045511 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5503f9-3d4f-4986-b5f6-4a135a00ca09 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:13.050635 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:13.051268 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:13.051542 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:13.051809 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:13.052448 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:13.054429 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:13.083340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1287/2585] 10.222.0.23 () {60 vars in 1153 bytes} [Wed Oct 12 02:42:13 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:42:13.087394 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:13.098562 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff781c-92ed-4e17-a8d9-002ba5899bb9 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:13.158830 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:13.159483 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:13.159761 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:13.160041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] PATH_INFO: `/v3/users` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:13.160722 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:13.162649 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:13.185242 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1299/2586] 10.222.0.23 () {60 vars in 1144 bytes} [Wed Oct 12 02:42:13 2022] POST /identity/v3/users => generated 295 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 12 02:42:13.190040 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:13.200881 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3a03e4-fc51-4483-8d71-25b8761a70f6 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:13.205774 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:13.206380 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:13.206659 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:13.206929 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:13.207580 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:13.209250 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:13.220834 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1288/2587] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:42:13.225181 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:13.236571 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e1907f-ea1e-4799-97eb-c5e24de0da91 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:13.241505 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:13.242169 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:13.242451 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:13.242850 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] PATH_INFO: `/v3/projects/499f753d52b945f7a5af4fddc75a9f10/users/3e986eab6ac04906b7b7a14647366d58/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:13.259825 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=499f753d52b945f7a5af4fddc75a9f10, user_id=3e986eab6ac04906b7b7a14647366d58, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:13.262566 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:13.276400 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1300/2588] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:42:13 2022] PUT /identity/v3/projects/499f753d52b945f7a5af4fddc75a9f10/users/3e986eab6ac04906b7b7a14647366d58/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:42:13.280478 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:13.291137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70271639-407f-48c1-971a-ff92c120c80d admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:13.351479 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95456f6-8703-48e9-870c-fec09a0ca84d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:13.352424 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95456f6-8703-48e9-870c-fec09a0ca84d admin admin] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:13.352740 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95456f6-8703-48e9-870c-fec09a0ca84d admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:13.353041 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95456f6-8703-48e9-870c-fec09a0ca84d admin admin] PATH_INFO: `/v3/roles` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:13.353713 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95456f6-8703-48e9-870c-fec09a0ca84d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:13.355389 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95456f6-8703-48e9-870c-fec09a0ca84d admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:13.366469 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1289/2589] 10.222.0.23 () {58 vars in 1122 bytes} [Wed Oct 12 02:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 12 02:42:13.370727 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:13.381535 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862a241e-69df-4a9d-ac47-b03baa637fd4 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:13.390901 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:13.391688 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] REQUEST_METHOD: `PUT` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:13.392035 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:13.392307 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] PATH_INFO: `/v3/projects/499f753d52b945f7a5af4fddc75a9f10/users/3e986eab6ac04906b7b7a14647366d58/roles/32a3720c5c9442649cd9a8c93b3bbbd7` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:13.394687 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] RBAC: Authorizing `identity:create_grant(project_id=499f753d52b945f7a5af4fddc75a9f10, user_id=3e986eab6ac04906b7b7a14647366d58, role_id=32a3720c5c9442649cd9a8c93b3bbbd7)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:13.397294 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:13.409983 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1301/2590] 10.222.0.23 () {60 vars in 1483 bytes} [Wed Oct 12 02:42:13 2022] PUT /identity/v3/projects/499f753d52b945f7a5af4fddc75a9f10/users/3e986eab6ac04906b7b7a14647366d58/roles/32a3720c5c9442649cd9a8c93b3bbbd7 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:42:18.050411 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f967a05-d74c-4144-9332-6cce1cdbb35a None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:18.051004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f967a05-d74c-4144-9332-6cce1cdbb35a None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:18.051004 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f967a05-d74c-4144-9332-6cce1cdbb35a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:18.087124 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-0f967a05-d74c-4144-9332-6cce1cdbb35a None None] MFA Rules not processed for user `3e986eab6ac04906b7b7a14647366d58`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:18.088438 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0f967a05-d74c-4144-9332-6cce1cdbb35a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:18.165051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1290/2591] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:18 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:18.185412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:18.187779 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:18.267356 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01afb429-7011-4012-91e0-910d0bebf357 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:18.273931 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:18.274595 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:18.274874 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:18.275553 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:18.282670 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:18.339017 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:18.340616 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:18.350182 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1302/2592] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:42:28.162138 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=94,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:28.345848 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=24 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:33.775816 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:33.776356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:33.776356 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:33.795486 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] MFA Rules not processed for user `e184afaa0abd4e8eb1bc641167dc0b9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:42:33.797349 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:33.804147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1291/2593] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:42:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:42:34.550389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:34.552780 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:34.640561 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f90d2-b880-49f4-87e7-7570bdc08ca3 service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:34.645517 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:34.646121 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:34.646395 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:34.646661 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:34.648487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:34.661469 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:34.663036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:34.673552 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1303/2594] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:42:43.790134 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=3 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:44.668647 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=32 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:42:47.587674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:47.589198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:47.607715 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840990a6-6fea-4d00-8439-b3d4a3fc81ec None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:47.613915 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4947f-b557-4216-854c-352348504853 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:47.614525 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4947f-b557-4216-854c-352348504853 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:47.614795 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4947f-b557-4216-854c-352348504853 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:47.615056 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4947f-b557-4216-854c-352348504853 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:47.616787 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-2af4947f-b557-4216-854c-352348504853 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:47.630609 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4947f-b557-4216-854c-352348504853 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:47.632198 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4947f-b557-4216-854c-352348504853 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:47.642845 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1292/2595] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:42:50.013456 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:50.067401 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c189563-625a-458a-bfba-f3a1898d6f41 service placement] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:50.071019 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:50.071617 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:50.071892 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:50.072163 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:50.081621 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:50.083036 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:50.092389 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1304/2596] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:50 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:42:51.843014 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4947f-b557-4216-854c-352348504853 service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:42:51.855433 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4947f-b557-4216-854c-352348504853 service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:42:51.861088 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:42:51.861695 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:42:51.861972 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:42:51.862242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:42:51.863961 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:42:51.877003 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:42:51.878589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:42:51.889290 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1293/2597] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:42:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:43:00.088172 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=26 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:43:01.884665 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=12 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:43:43.685227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:43:43.686683 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:43:43.701061 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cec28e-1aad-4e6e-a6d6-a868d1723fc3 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:43:43.706475 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:43:43.707065 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:43:43.707340 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:43:43.707605 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:43:43.709306 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:43:43.720746 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:43:43.722884 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:43:43.732912 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1305/2598] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:43:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:43:53.728674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=6 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:44:46.132998 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:44:46.134466 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:44:46.146242 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd2d64c-46f3-467c-9a87-f4483237d2a3 service nova] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:44:46.149723 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:44:46.150298 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:44:46.150568 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:44:46.150831 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:44:46.152462 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:44:46.165877 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:44:46.167279 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1633a6e6-871f-4f59-83fa-579eb926efab service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:44:46.177988 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1294/2599] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:44:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:44:47.354720 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:44:47.363926 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec3001c-17f0-4b95-913d-e7b2f9f4ba9a service placement] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:44:47.367435 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:44:47.368074 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:44:47.368295 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:44:47.368561 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:44:47.370457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:44:47.383777 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:44:47.385212 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:44:47.396319 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1306/2600] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:44:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:44:49.851255 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] REQUEST_METHOD: `POST` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:44:49.851704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:44:49.851801 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:44:49.871601 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.auth.core [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] MFA Rules not processed for user `e184afaa0abd4e8eb1bc641167dc0b9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81365) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:44:49.872826 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:44:49.880401 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1295/2601] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:44:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:44:50.191405 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:44:50.200078 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4af30d1-fbba-4b46-b11d-ccf0a0f096a5 service neutron] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:44:50.203673 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:44:50.204261 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:44:50.204532 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:44:50.204835 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:44:50.206531 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:44:50.215766 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:44:50.217213 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a6f52c-32d0-486a-a7b3-5b08c61c4e29 service neutron] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:44:50.225657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1307/2602] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:44:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:44:59.866716 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=9 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:45:00.221468 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=12 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:45:38.063053 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:45:38.073289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb893e9-a110-4040-a583-a898e0cad9d9 None None] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:45:38.076870 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:45:38.077460 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:45:38.077733 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:45:38.078000 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:45:38.079704 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:45:38.090206 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:45:38.091702 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:45:38.106663 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1296/2603] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:45:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:45:48.096396 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:46:40.936022 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] REQUEST_METHOD: `POST` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:46:40.936537 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:46:40.936641 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:46:40.955217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.auth.core [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] MFA Rules not processed for user `e184afaa0abd4e8eb1bc641167dc0b9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81364) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 12 02:46:40.956403 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:46:40.963217 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1308/2604] 10.222.0.23 () {58 vars in 958 bytes} [Wed Oct 12 02:46:40 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 12 02:46:41.282443 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:46:41.291674 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06820bde-051d-4b63-9572-d5fde93f506c service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:46:41.295310 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:46:41.295890 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:46:41.296159 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:46:41.296422 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:46:41.298127 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:46:41.307590 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:46:41.309033 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:46:41.316936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1297/2605] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:46:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:46:44.175153 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:46:44.190914 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76591649-e6ca-41ed-8538-873f63ae5e32 None None] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:46:44.197792 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:46:44.198412 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:46:44.198737 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:46:44.199006 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:46:44.200846 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:46:44.214863 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:46:44.217096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:46:44.230236 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1309/2606] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:46:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:46:51.313451 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:46:54.223674 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=9 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:48:30.113745 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:30.133169 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8534ac6d-0ebc-4bda-a1bb-ca252c78dad2 service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:30.136790 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:30.137385 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:30.137660 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:30.137927 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:30.160345 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:30.161821 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:30.170544 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1298/2607] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:48:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:48:33.473229 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:33.517013 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba4046f-2a63-4563-862f-43809b2ad055 service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:33.522749 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa015e7713c34f16be4529081c849ccf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:33.524748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:33.524748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:33.524748 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:33.533593 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:33.535488 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:33.549773 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1310/2608] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:48:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:48:40.166856 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=17 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:48:43.116147 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:43.125104 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150c626-0111-44e9-96d4-9bc6ee280d6b service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:43.128735 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8c5cd3df4be042f7ac338b9a6b0a50da', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:43.129318 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:43.129598 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:43.129867 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:43.198936 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:43.200372 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:43.219289 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1299/2609] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:48:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:48:43.544907 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=20 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:48:45.997272 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:46.042672 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd1e9fc-a33d-4776-a28a-c9642ec3ef53 service nova] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:46.049482 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:46.050114 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:46.050409 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:46.050702 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:46.052457 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:48:46.109768 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:46.111771 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:46.124240 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1311/2610] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:48:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:48:49.461940 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:49.473154 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f5894d-0569-4f6f-b1d5-a25da2d898af service neutron] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:49.478194 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:49.478819 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:49.479095 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:49.479364 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:49.481505 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:48:49.494032 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:49.496024 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:49.515225 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1300/2611] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:48:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:48:49.520041 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:49.531440 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08607d60-caf8-4c8b-8192-549f43322737 service placement] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:49.536034 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:49.536657 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:49.536956 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:49.537221 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] PATH_INFO: `/v3/users/e184afaa0abd4e8eb1bc641167dc0b9b` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:49.538328 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e184afaa0abd4e8eb1bc641167dc0b9b)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:49.540292 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:49.587073 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e184afaa0abd4e8eb1bc641167dc0b9b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.598744 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e184afaa0abd4e8eb1bc641167dc0b9b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.599406 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e184afaa0abd4e8eb1bc641167dc0b9b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.606586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e184afaa0abd4e8eb1bc641167dc0b9b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.607227 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e184afaa0abd4e8eb1bc641167dc0b9b'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.619542 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1312/2612] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:48:49 2022] DELETE /identity/v3/users/e184afaa0abd4e8eb1bc641167dc0b9b => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:48:49.790016 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:49.801116 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38f5109-c694-4f08-b979-265b2ddaaa8c service placement] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:49.863137 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:49.864012 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:49.864285 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:49.864556 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] PATH_INFO: `/v3/projects/0420b1fe8ff5451dba85de3001a3932f` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:49.865738 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0420b1fe8ff5451dba85de3001a3932f)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:49.867680 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:49.917849 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0420b1fe8ff5451dba85de3001a3932f', 'request_id': 'req-37542919-2a92-4a69-a781-7059f61fad74'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.923956 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0420b1fe8ff5451dba85de3001a3932f', 'request_id': 'req-37542919-2a92-4a69-a781-7059f61fad74'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.930808 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0420b1fe8ff5451dba85de3001a3932f', 'request_id': 'req-37542919-2a92-4a69-a781-7059f61fad74'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.933534 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:48:49.961051 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] The token cache is being invalidate because project 0420b1fe8ff5451dba85de3001a3932f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:48:49.961355 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.961974 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:49.963442 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1301/2613] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:48:49 2022] DELETE /identity/v3/projects/0420b1fe8ff5451dba85de3001a3932f => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:48:49.967442 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:49.968902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:48:49.982106 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4394a6d3-a058-4ab4-b93c-d944b36df549 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:50.037586 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:50.038216 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:50.038487 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:50.038762 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] PATH_INFO: `/v3/users/3e986eab6ac04906b7b7a14647366d58` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:50.039879 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e986eab6ac04906b7b7a14647366d58)` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:50.041814 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:50.069852 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:48:50.089070 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e986eab6ac04906b7b7a14647366d58'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.100758 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e986eab6ac04906b7b7a14647366d58'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.101433 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e986eab6ac04906b7b7a14647366d58'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.108990 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e986eab6ac04906b7b7a14647366d58'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.109600 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.notifications [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e986eab6ac04906b7b7a14647366d58'} {{(pid=81364) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.122300 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1313/2614] 10.222.0.23 () {58 vars in 1224 bytes} [Wed Oct 12 02:48:49 2022] DELETE /identity/v3/users/3e986eab6ac04906b7b7a14647366d58 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 12 02:48:50.292650 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:50.294117 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:48:50.308589 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37542919-2a92-4a69-a781-7059f61fad74 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:50.365415 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1bca8e4c760d498693a8dcfc0e178249', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b152cd4fded41fa8ba95182a5dc98e4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:50.366061 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:50.366333 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:50.366610 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] PATH_INFO: `/v3/projects/499f753d52b945f7a5af4fddc75a9f10` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:50.367726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] RBAC: Authorizing `identity:delete_project(project_id=499f753d52b945f7a5af4fddc75a9f10)` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:50.369682 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:50.415023 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '499f753d52b945f7a5af4fddc75a9f10', 'request_id': 'req-0230f6ff-8acd-45c4-a8d1-36c25848f560'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.420487 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '499f753d52b945f7a5af4fddc75a9f10', 'request_id': 'req-0230f6ff-8acd-45c4-a8d1-36c25848f560'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.427283 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '499f753d52b945f7a5af4fddc75a9f10', 'request_id': 'req-0230f6ff-8acd-45c4-a8d1-36c25848f560'} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.440340 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] The token cache is being invalidate because project 499f753d52b945f7a5af4fddc75a9f10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81365) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 12 02:48:50.440625 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.441270 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.notifications [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81365) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 12 02:48:50.442736 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1302/2615] 10.222.0.23 () {58 vars in 1233 bytes} [Wed Oct 12 02:48:50 2022] DELETE /identity/v3/projects/499f753d52b945f7a5af4fddc75a9f10 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 12 02:48:50.678096 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Authenticating user token {{(pid=81364) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:48:50.679573 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:48:50.749752 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672a71e7-b1a3-494e-b5ce-78f18800d655 admin admin] Validating token access rules against request {{(pid=81364) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:48:50.810800 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81364) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:48:50.811414 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] REQUEST_METHOD: `GET` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:48:50.811736 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] SCRIPT_NAME: `/identity` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:48:50.812083 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81364) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:48:50.813902 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.fernet_utils [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81364) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:48:50.826724 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:48:50.828373 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be257cc0-f087-4010-bb08-8c1be2298079 service placement] RBAC: Authorization granted {{(pid=81364) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:48:50.839711 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: [pid: 81364|app: 0|req: 1314/2616] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:48:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 12 02:49:00.437434 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:DELETE=8,keystone:SELECT=58,keystone:INSERT=1 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:49:00.835072 dv-cinder-849143-14-iscsi devstack@keystone.service[81364]: DEBUG dbcounter [-] [81364] Writing DB stats keystone:DELETE=6,keystone:SELECT=77,keystone:INSERT=1 {{(pid=81364) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 12 02:53:53.651726 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Authenticating user token {{(pid=81365) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 12 02:53:53.653176 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.fernet_utils [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81365) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 12 02:53:53.668045 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0230f6ff-8acd-45c4-a8d1-36c25848f560 admin admin] Validating token access rules against request {{(pid=81365) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 12 02:53:53.673112 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a5d1a9-b844-444f-9bbf-56b954e74122 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '38f000e12e7b4cd1b34875f6c0bf9cc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0e6c7a0acc5d40ffb16cb54d7c51720f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81365) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 12 02:53:53.673657 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5d1a9-b844-444f-9bbf-56b954e74122 service placement] REQUEST_METHOD: `GET` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 12 02:53:53.673931 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5d1a9-b844-444f-9bbf-56b954e74122 service placement] SCRIPT_NAME: `/identity` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 12 02:53:53.674199 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a5d1a9-b844-444f-9bbf-56b954e74122 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81365) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 12 02:53:53.682839 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a5d1a9-b844-444f-9bbf-56b954e74122 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 12 02:53:53.684408 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a5d1a9-b844-444f-9bbf-56b954e74122 service placement] RBAC: Authorization granted {{(pid=81365) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 12 02:53:53.694719 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: [pid: 81365|app: 0|req: 1303/2617] 10.222.0.23 () {60 vars in 1365 bytes} [Wed Oct 12 02:53:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 12 02:54:03.689986 dv-cinder-849143-14-iscsi devstack@keystone.service[81365]: DEBUG dbcounter [-] [81365] Writing DB stats keystone:SELECT=6 {{(pid=81365) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}